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Abstract 

This  research  studied  the  development  of  source  selec¬ 
tion  evaluation  criteria  and  standards  for  reliability  and 
maintainability.  The  data  base  consisted  of  information 
obtained  during  personal  interviews  with  personnel  from  Air 
Force  Systems  Command,  Aeronautical  Systems  Division  and 
the  Air  Force  Acquisition  Logistics  Center.  Those 
interviewed  were  experienced  in  the  development  of  source 
selection  criteria  and  standards  for  reliability  and 
maintainability.  - - - 

The  research  culminated  in  the  establishment  of  a 
guide  for  the  development  of  source  selection  evaluation 
criteria  and  standards  for  reliability  and  maintainability. 
The  guide  is  not  specific  to  a  particular  type  of  system 
or  phase  in  the  acquisition  process.  The  guide  provides 
general  procedures  and  areas  of  consideration  for  develop¬ 
ment  of  criteria  and  standards  which  may  be  applied  to 
various  acquisitions. 


DEVELOPING  SOURCE  SELECTION  EVALUATION  CRITERIA  AND 
STANDARDS  FOR  RELIABILITY  AND  MAINTAINABILITY 

I.  Research  Proposal 


Introduction 

j  Reliability  and  Maintainability  (R&M)  issues  have 
become  the  prime  focus  of  attention  within  the  Air  Force  in 
the  development  and  acquisition  of  major  weapon  systems. 

R&M  considerations  must  be  continuously  addressed  to  insure 
readiness  of  our  Air  Force.  A  weapon  system  must  be  able 
to  perform  with  consistent  reliability  and  be  designed  for 

efficient  and  effective  maintainability.  - 

The  primary  document  initiating  the  focus  on  R&M  is 
an  action  memorandum  signed  on  17  September  1984  by  Chief 
of  Staff  Gen  Charles  A  Gabriel  and  Secretary  of  the  Air 
Force  Verne  Orr.  The  memorandum  stresses  the  importance  of 
considering  R&M  in  the  acquisition  process.  As  stated  in 
the  memorandum. 


For  too  long,  the  reliability  and  maintain¬ 
ability  of  our  weapon  systems  have  been 
secondary  considerations  in  the  acquisition 
process.  It  is  time  to  change  this  practice  and 
make  reliability  and  maintainability  primary 
considerations  [21:1]. 


The  memorandum  emphasizes  considering  R&M  throughout  the 
acquisition  process.  As  stated  in  the  memorandum. 


We  must  emphasize  reliability  and  maintain¬ 
ability  throughout  the  acquisition  process  — 
from  requirement  definition,  through  concept 
development,  design,  production,  and  acceptance. 
Everyone  must  insure  reliability  and  maintain¬ 
ability  requirements  are  met  through  every  step 
of  the  process.  Reliability  and  maintainability 
must  be  coequal  with  cost,  schedule,  and  perfor¬ 
mance  as  we  bring  a  system  into  the  Air  Force 
inventory  [21:1]. 


The  source  selection  process  is  the  avenue  through 
which  major  weapon  systems  are  brought  into  the  Air  Force 
inventory.  The  major  criteria  used  in  the  source  selection 
process  have  historically  been  cost,  schedule,  and  perfor¬ 
mance.  As  quoted  above,  R&M  must  be  coequal  with  these 
criteria. 


Problem  Statement 

Proper  evaluation  of  R&M  in  the  source  selection 
process  is  imperative  to  insure  the  most  reliable  and  main¬ 
tainable  weapon  systems  are  acquired,  however,  guidance  is 
dispersed  as  to  methods  of  properly  establishing  R&M  source 
selection  criteria  and  standards. 

McLennan  in  his  study  of  source  selection  evaluation 
criteria  stated  that,  "A  complicating  factor  .  .  .  (in 
selecting  the  most  appropriate  criteria)  ...  is  that  the 
personnel  employed  on  source  selections  often  have  little 
previous  source  selection  experience  and  may  be  approaching 
a  difficult  task  with  but  a  vague  sense  of  direction" 

(6:4).  With  the  advent  of  R&M  2000  and  the  new  emphasis  of 
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R&M  as  coequal  with  cost,  schedule,  and  performance  in 
source  selection,  the  complications  stated  by  McLennan 
become  greater. 


A  study  of  current  guidance  for  establishing  and 
applying  R&M  source  selection  evaluation  criteria  and 
standards  is  needed  to  provide  personnel  with  direction  as 
to  effective  evaluation  of  R&M  in  source  selection. 


Background 

Increased  emphasis  within  the  U.S.  Air  Force  on  the 
issue  of  reliability  and  maintainability  considerations  has 
directly  impacted  the  major  weapon  system  acquisition 
process.  Excerpts  previously  stated  from  the  17  September 
1984  action  memorandum  confirm  this  emphasis.  Also  stated 
in  the  memorandum  is  the  development  of  an  Air  Force  wide 
action  plan  on  R&Ms 


To  institutionalize  the  Air  Force  commitment,  Lt 
Gen  Bob  Russ  and  Lt  Gen  Marquez  are  forming  a 
group  of  logisticians,  operators,  and  acquisi¬ 
tion  specialists  to  develop  an  Air  Force-wide 
action  plan  with  specific  recommendations  and 
suspenses.  This  will  be  reported  to  us  in  early 
December  [21:1]. 


On  1  February  1985,  the  Reliability  and  Maintain¬ 
ability  Action  Plan  R&M  2000  was  approved  (20).  The  plan 
contains  a  multitude  of  actions  to  be  accomplished  to 
insure  R&M  issues  are  considered  throughout  the  Air  Force. 


The  action  plan  serves  as  further  justification  for 
the  importance  of  this  thesis  in  providing  guidance  on  the 
effective  use  of  R&M  evaluation  criteria  and  standards. 

The  plan  states ,  "The  action  plan  is  aimed  at  ensuring  R&M 
is  considered  across  all  of  our  weapon  systems  and  treated 
equally  with  cost,  schedule,  and  performance"  (20: i). 

One  of  the  six  key  management  objectives  in  the 
action  plan  is  to,  "Establish  industry  commitment  to  R&M  to 
ensure  contractors  have  the  motivation  and  capability  to 
support  Air  Force  R&M  requirements"  (20:i).  This  objective 
can  be  obtained  only  through  proper  identification  of  R&M 
requirements.  These  requirements  are  communicated  in  the 
form  of  R&M  evaluation  criteria  which  are  contained  in  the 
Request  for  Proposal  (RFP) . 

To  ensure  R&M  is  properly  addressed  in  weapon  system 
acquisitions,  the  action  plan  calls  for  the  following: 

Review  selected  weapon  systems  decision  documen¬ 
tation  including  the  statement  of  need,  program 
management  directive,  decision  coordinating 
paper,  request  for  proposal,  source  selection 
plan,  acquisition  plan,  and  R&M  plan  on  weapon 
system  programs  to  ensure  R&M  is  adequately 
addressed  and  to  assess  their  impact  on  opera¬ 
tional  support  [20:7]. 

A  further  action  which  directly  relates  to  this 
thesis  is  as  follows: 

Publish  guidebooks  for  implementing  successful 
R&M  programs  on  new  and  fielded  systems  to 
provide  a  source  of  R&M  information  for  program 


managers,  system  program  managers,  engineers, 
and  other  R&M  functional  personnel  [20:9]. 


Adding  further  impetus  to  the  importance  of  proper 
R&M  Evaluation  criteria  and  standards,  the  action  plan 
states  the  following: 


Contractors  design,  develop,  and  manufacture 
weapon  systems  in  response  to  the  requirements 
and  priorities  expressed  in  requests  for 
proposals  and  contracts.  The  specification, 
statement  of  work,  and  proposal  evaluation 
factors  are  clear  expressions  of  the  level  of 
the  Air  Force  commitment  and  priority  for  R&M. 
Accelerated  improvements  in  R&M  can  only  be 
attained  if  these  documents  clearly  communicate 
and  reinforce  the  R&M  commitment  [20:11]. 


As  a  final  action  relating  to  this  thesis,  R&M  2000 
requires  the  following: 


Increase  consideration  of  R&M  in  all  weapon 
system  source  selections  and  include  R&M  exper¬ 
tise  in  source  selection  organizations  to  ensure 
the  Air  Force  commitment  to  R&M  is  reflected  in 
contract  awards  [20:11]. 


General  Skantze,  Commander,  Air  Force  Systems 
Command,  in  a  letter  dated  21  December  1984,  established 
command-wide  policy  emphasizing  reliability,  maintain¬ 
ability,  and  producibility  in  the  design  process.  General 
Skantze  stated  that,  "The  operational  utility  of  our 
systems  is  fundamentally  rooted  in  how  well  we  have  inte¬ 
grated  reliability,  maintainability,  and  producibility  in 
the  design  process"  (11:1).  To  implement  this  focus  in 


the  acquisition  process.  General  Skantze  stated  that,  . 
.  we  are  changing  the  source  selection  process  criteria  to 
add  that  emphasis  and  incentivize  the  contractor  to  focus 
his  talents  on  this  goal”  (11:1).  Specific  implementation 
of  policy  in  the  source  selection  process  is  as  follows: 


He  will  emphasize  reliability,  maintainability, 
and  producibility  in  the  source  selection 
process  to  a  higher  degree  than  ever  before.  As 
part  of  the  source  selection  process,  you  will 
single  out  reliability,  maintainability,  and 
producibility  of  the  design  as  specific  evalua¬ 
tion  criteria  items.  Traditionally,  we  have 
operational  suitability  as  the  most  important 
area,  but  without  a  corresponding  ranking  of 
these  critical  design  parameters  as  specific 
items  within  that  area  [11:1]. 


General  Skantze  emphasizes  that  in  a  source  selection  for  a 
new  design,  design  for  reliability,  maintainability,  and 
producibility  are  to  be  ranked  as  the  first  items  in  the 
highest  ranked  area  (11:1) .  Offerors  are  also  to  be 
instructed  to  demonstrate  thoroughly  and  precisely  in  their 
proposals  how  reliability,  maintainability,  and  produci¬ 
bility  are  weighed  in  the  design  of  their  systems  (11:1). 

The  most  recent  research  at  hand  dealing  with  source 
selection  criteria  is  an  Air  Force  Institute  of  Technology 
thesis  completed  in  September  1984  entitled.  The  Feasi¬ 
bility  of  a  Decision  Support  System  for  the  Determina¬ 
tion  of  Source  Selection  Evaluation  Criteria  (6) . 

Although  the  thesis  does  not  specifically  address  R&M 


criteria  in  source  selection,  the  scope  of  the  thesis  is 
aimed  at  improving  the  source  selection  process  through 
meaningful  source  selection  evaluation  criteria.  Portions 
of  the  thesis  research  are  applicable  to  the  issue  of  R&M 
criteria.  As  stated  in  the  thesis  by  McLennan, 

Inappropriately  selected  criteria  renders  the 
evaluation  procedure  invalid  before  it  even 
starts,  and  an  invalid  source  selection  is  not 
only  very  costly  and  time-wasting,  but  may  be 
catastrophic  for  the  entire  project.  Thus  the 
selection  of  the  most  suitable  criteria  for  a 
given  acquisition  is  a  vital  element  of  the 
source  selection  process  [6:4]. 

Reliability  and  maintainability  are  critical  factors 
to  be  considered  in  major  weapon  system  acquisitions  and 
more  specifically,  as  source  selection  evaluation  criteria. 
The  need  exists  for  identification  of  critical  require¬ 
ments  within  R&M  that  must  be  addressed  as  source  selection 
criteria  and  used  to  establish  source  selection  standards. 
Assimilating  the  R&M  factors  into  a  readily  available  docu¬ 
ment  will  serve  to  make  the  source  selection  process  more 
efficient  as  personnel  will  have  a  useful  guide  to  aid  in 
establishing  evaluation  criteria  and  standards.  McLennan 
emphasizes  the  difficulty  in  establishing  criteria  as  he 
states,  "The  selection  of  evaluation  criteria  is  a  particu¬ 
larly  difficult  job  for  the  inexperienced  and  a  potential 
source  of  future  problems  for  the  source  selection  process 
and  the  program"  (6:5). 


Scope  of  the  Research 

Through  analysis  of  current  guidance  and  interviews 
with  personnel  involved  in  R&M  within  Aeronautical  Systems 
Division  (ASD)  and  the  Air  Force  Acquisition  Logistics 
Center  (AFALC) ,  this  thesis  will  provide  insight  into 
effective  means  of  establishing  R&M  source  selection 
evaluation  criteria  and  standards. 

R&M  evaluation  criteria  and  standards  must  be 
tailored  to  each  acquisition  situation  based  on  the 
complexity  of  the  weapon  system  being  acquired  and  the 
acquisition  process  phase.  The  four  basic  phases  of  the 
acquisition  process  are:  (1)  concept  exploration,  (2) 
demonstration/ validation,  (3)  full  scale  development,  and 
(4)  production. 

R&M  requirements  versus  acquisition  process  phases 
were  discussed  during  a  personal  interview  with  Mr.  Clay 
Nieman,  R&M  Engineer,  Inertial  Navigation  Unit  Program 
office  (7).  Mr.  Nieman  provided  a  general  relationship 
between  R&M  requirements  and  the  particular  phase  in  the 
acquisition  process.  R&M  requirements  are  minimal  in  the 
conceptual  phase  as  the  system  design  is  in  preliminary 
stages.  Requirements  increase  during  the  demonstration/ 
validation  phase  and  peak  at  full  scale  development  as  the 
system  design  is  completed.  R&M  requirements  tend  to 
slightly  decrease  during  the  production  phase. 


Tables  1  and  II,  Application  Matrix  for  Reliability 
and  Application  Matrix  for  Maintainability,  provide 
personnel  involved  in  RFP  preparation  with  guidance  as  to 
reliability  and  maintainability  tasks  which  should  be 
included  during  various  stages  of  acquisition.  The  parti¬ 
cular  tasks  selected  will  depend  on  the  system  being 
acquired  and  phase  of  acquisition.  The  tables  serve  to 
illustrate  the  varying  degree  of  reliability  and  maintain¬ 
ability  requirements  during  different  stages  of  acquisi¬ 
tion.  As  can  be  seen,  few  requirements  are  applicable 
during  concept  exploration  while  at  full  scale  development, 
generally  all  tasks  are  applicable.  R&M  tasking  decreases 
during  production. 

Although  R&M  requirements  vary  for  different  types  of 
systems  and  during  various  stages  of  acquisition,  this 
thesis  will  provide  general  guidance  which  may  be  applied 
to  various  acquisition  situations. 

R&M  requirements  have  been  in  existence  in  one  form 
or  another  since  the  development  of  the  first  weapon 
system.  Today,  explicit  guidance  exists  which  specifies 
various  R&M  parameters  and  government  specifications  which 
must  be  built  into  each  weapon  system.  For  the  first  time, 
however,  R&M  is  being  stressed  as  the  number  one  considera¬ 
tion  within  the  technical  area  in  source  selections.  In 
recognition  of  the  difficulty  in  implementing  such  a 
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TABLE  I 


Reliability  Application  Matrix  (16:A-1) 


PROGRAM  PHASE 

1  MwM 

TTPE 

CONCEPT 

VALID 

FSED 

PROD 

101 

htlMUIUTr  PROGRAM  PUN 

MGT 

s 

3 

G 

C 

102 

MONITOR/ CONTROL  OK  SUBCONTRACTORS 

AND  SUPPLIERS 

MGT 

S 

S 

G 

Q 

103 

PROGRAM  REVIEWS 

MGT 

s 

s(2> 

0(2) 

0(2) 

104 

FAILURE  REPORTINC,  ANALYSIS,  AND 
CORRECTIVE  ACTION  SYSTEM  (FRACAS) 

ENG 

HA 

s 

G 

C 

105 

FAILURE  REVIEW  BOARD  (FRB) 

MGT 

HA 

s(2> 

G 

G 

201 

RELIABILITY  MODELING 

ENG 

s 

S(2) 

0(2) 

CC(2> 

202 

RELIABILITY  ALLOCATIONS 

ACC 

s 

G 

G 

GC 

203 

RELIABILITY  PREDICTIONS 

ACC 

s 

S(2) 

G(2) 

CC(2) 

204 

FAILURE  MODES,  EFFECTS,  AND 
CRITICALITY  ANALYSIS  (FMECA) 

ENG 

S 

S 

( 1 ) (2) 

G 

( 1)  (2) 

GC 

(1)(2) 

205 

SNEAE  CIRCUIT  ANALYSIS  (SCA) 

ENG 

NA 

NA 

(XI) 

GC(  1 ) 

206 

ELECTRONIC  PART Sy CIRCUITS 

TOLERANCE  ANALYSIS 

ENG 

NA 

NA 

G 

GC 

207 

PARTS  PROGRAM 

ENG 

S 

s(2>(3> 

C<2) 

0(2) 

208 

RELIABILITY  CRITICAL  ITEMS 

MGT 

S(1) 

S(l) 

G 

C 

209 

EFFECTS  OF  FUNCTIONAL  TESTING. 

STORAGE,  HANDLING,  PACKAGING, 
TRANSPORTATION.  AND  MAINTENANCE 

ENG 

NA 

S(1> 

G 

GC 

301 

ENVIRONMENTAL  STRESS  SCREENING 
(ESS) 

ENG 

NA 

S 

G 

0 

302 

RELIABILITY  DCVELOPHENT/CROWTH 
TESTING 

ENG 

NA 

5(2) 

G(2) 

NA 

303 

RELIABILITY  QUALIFICATION  TEST 
(ROT)  PROGRAM 

ACC 

NA 

'  S(2) 

0(2) 

0(2) 

304 

PRODUCTION  RELIABILITY  ACCEPTANCE 
ACCEPTANCE  TEST  (PRAT)  PROGRAM 

ACC 

NA 

NA 

s 

0(2)(3) 

COBB  flfcf IIUTlQIli'i 


xiat  ,itp6=  rnoaiUH  riuss 


»CC  -  RELIABILITY  ACCOUNTING  S  -  SELECTIVELY  APPLICABLE 

ENC  -  RELIABILITY  ENGINEERING  G  -  GENERALLY  APPLICABLE 

HOT  -  MANAGEMENT  OC  .  GENERALLY  APPLICABLE  TO  DESIGN 

CHANGES  ONLY 

NA  .  NOT  APPLICABLE 

(1)  -  REQUIRES  CONSIDERABLE  INTERPRETATION 

OP  INTENT  TO  BE  COST  EFFECTIVE 

(2)  -  MIL-STD-T85  IS  NOT  THE  PRIMARY 

IMPLEMENTATION  REQUIREMENT.  OTHER 
MIL.STDS  OR  STATEMENT  OF  WORE 
REQUIREMENTS  MUST  BE  INCLUDED  TO 
DEFINE  THE  REQUIREMENTS. 
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TABLE  II 


Maintainability  Application  Matrix  (12:A-1) 


TASK  TITLE 

TASK 

TYPE 

PROGRAM  PHASE 

CON¬ 

CEPT 

VALID 

FSO 

PROO 

OPERAT  SYSTEM 
OEV  (MOOS) 

101 

Maintainability  Program 

Plan 

MGT 

N/A 

8(3) 

G 

6(3) (1) 

GO) 

102 

Monitor/Control  of  Sub¬ 
contractors  and  Vendors 

MGT 

N/A 

S 

6 

6 

S 

103 

Program  Reviews 

MGT 

s 

6(3) 

6 

6 

s 

104 

Data  Collection, 

Analysis  and  Corrective 
Action  System 

ENG 

N/A 

S 

6 

G 

s 

201 

Maintainability  Modeling 

ENG 

S 

s<«) 

G 

C 

N/A 

202 

Maintainability 

A1  locations 

ACC 

s 

S(4)  . 

G 

C 

5(4) 

203 

Maintainability 

Predictions 

ACC 

N/A 

S(2> 

6(2) 

C 

5(2) 

204 

Failure  Nodes  and 

Effects  Analysis  (FMEA) 
Maintainability  Information 

ENG 

N/A 

S(2) 

(3) (4) 

6(1) 

(2) 

C(l) 

(2) 

5(2) 

20$ 

Maintainability  Analysts 

ENG 

5(3) 

6(3) 

6(1) 

C(l) 

5 

206 

Maintainability  Design 
Criteria 

ENG 

N/A 

3(3) 

G 

C 

5 

207 

Preparation  of  Inputs  to 
Detailed  Maintenance  Plan 
and  logistics  Support 
Analysis  (LSA) 

ACC 

N/A 

S[2) 

(3) 

6(2) 

C(2) 

5 

301 

Maintainability 
Demonstration  (MO) 

ACC 

N/A 

5(2) 

6(2) 

C  ( 2) 

5(2) 

coot  DEFINITIONS 
S  -  Selectively  applicable 
G  -  Generally  Applicable 

C  •  Generally  Applicable  to  design  changes  only 

n/a  -  not  applicable 

ACC  -  Maintainability  Accounting 

ENG  ■  Maintainability  Engineering 


MGT  -  Management 


(1)  Requires  considered  Interpretation  of  intent  to  be  cost  effective. 


(2) 


MIL-STD-470  is  not  the  primary  lap  lamentation  dociment.  Other  NIl-STDS  or  Statement  of 
reoutrcMflts  mU  b#  induced  to  define  or  rtsclnd  tne  pnulrmwti.  For  e*i^>ie 
ssist  be  imposed  to  describe  maintainability  dmeonstratlon  details  and  methods. 


(3)  Appropriate  for  those  task  elements  suitable  to  definition  during  phase. 

(A)  Oepends  on  physical  complealty  of  the  system  unit  being  procured,  its  packaging  and  Its  overall 
maintenance  policy. 
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requirement,  this  thesis  will  help  insure  that  proper  evalu 
ation  criteria  and  standards  are  established  so  that  the 
most  reliable  and  maintainable  systems  are  acquired. 

Summary  and  Preview 

This  chapter  has  provided  the  justification  for  the 
research  effort  to  include  an  introduction  to  the  subject/ 
problem  statement,  background  of  the  problem,  and  finally, 
the  scope  of  the  research. 

Chapter  II  will  provide  a  literature  review  of  the 
source  selection  process  to  include  applicable  regulations, 
source  selection  objectives,  source  selection  organization, 
the  source  selection  plan,  evaluation  criteria,  and  evalua¬ 
tion  standards.  Also  included  in  the  literature  review  is 
an  overview  of  reliability  and  maintainability  as  it 
applies  in  source  selection. 

Chapter  III  will  be  the  methodology  used  in  the 
research.  The  chapter  will  include  the  research  popula¬ 
tion,  data  base,  limitations  to  the  data  base,  the  research 
instrument,  and  data  analysis.  Chapter  IV  provides  the 
findings  obtained  based  on  examination  of  data.  Chapter  V 
provides  conclusions  and  recommendations  for  further 


research. 


II.  Literature  Review 


Introduction 

This  chapter  will  provide  an  overview  of  the  source 
selection  process  through  a  review  of  appropriate  Depart¬ 
ment  of  Defense  and  United  States  Air  Force  publications. 
This  chapter  will  also  include  an  overview  of  R&M  with 
emphasis  on  the  application  of  R&M  in  source  selection. 

Applicable  Regulations 

Department  of  Defense  Directive  (DODD)  5000.1,  Major 
Systems  Acquisition  Procedures  (14)  and  Department  of 
Defense  Instruction  5000.2,  Major  Systems  Acquisition 
Procedures  (13)  provide  mandatory  policy  in  regards  to 
contracting  for  major  weapon  systems.  Source  selection 
policy  is  guided  by  DODD  4105.62,  Selection  of  Contractual 
Sources  for  Major  Defense  Systems  (17).  These  documents 
discuss  the  acquisition  process  and  source  selection 
process  at  a  general  level.  Guidance  at  a  more  operational 
level  is  contained  in  Air  Force  Regulation  (AFR)  70-15, 
Source  Selection  Policy  and  Procedures  (23).  AFR  70-15 
will  be  the  main  source  of  reference  for  the  literature 
review  on  source  selection.  Applicable  Air  Force  Systems 
Command  (AFSC)  and  Aeronautical  Systems  Division  (ASD) 
supplements  to  AFR  70-15  along  with  ASD  guides  to  source 


selection  will  also  be  utilized  due  to  current  guidance 
being  issued  emphasizing  the  application  of  R&M  in  source 
selection. 


A  multitude  of  regulations  exist  detailing  R&M 
policies,  guidelines,  and  procedures.  In  order  to  provide 
a  broad  overview  of  R&M  and  how  it  applies  in  source  selec 
tion,  DODD  5000.40,  Reliability  and  Maintainability  (15), 
and  APR  800-18,  Air  Force  Reliability  and  Maintainability 
(18)  will  be  the  main  sources  of  reference  for  the  litera¬ 
ture  review  of  R&M. 


Source  Selection  Objectives 

DODD  4105.62  states  the  three  primary  objectives  of 
the  formal  source  selection  process  as  follows  (17:2): 


(a)  select  the  source  whose  proposal  has  the 
highest  degree  of  realism  and  credibility 
and  whose  performance  is  expected  to  best 
meet  Government  objectives  at  an  affordable 
cost; 

(b)  assure  impartial,  equitable,  and  comprehen¬ 
sive  evaluation  of  competitors'  proposals 
and  related  capabilities;  and 

(c)  maximize  efficiency  and  minimize  complexity 
of  solicitation,  evaluation  and  the  selec¬ 
tion  decision. 


The  principal  objective  of  the  source  selection 
process  a3  stated  in  APR  70-15  is. 


.  .  .  to  select  the  source  whose  proposal  has 
the  highest  degree  of  credibility  and  whose 
performance  can  be  expected  to  best  meet  the 
government's  requirements  at  an  affordable  cost. 


The  process  must  provide  an  impartial,  equit 
able,  and  comprehensive  evaluation  of  compet 
itor's  proposals  and  related  capabilities 
[23:3]. 

Source  Selection  Organization 

Figure  1  provides  the  overall  source  selection  organi 
zation.  The  official  designated  to  direct  the  source  selec 
tion  process  and  make  the  source  selection  decision  is  the 
Source  Selection  Authority  (SSA).  The  SSA  is  responsible 
for  proper  and  efficient  conduct  of  the  process  to  include 
proposal  solicitation,  evaluation,  selection,  and  contract 
award  (23:5). 

The  SSA  is  advised  by  the  Source  Selection  Advisory 
Council  (SSAC).  The  SSAC  provides  the  SSA  with  a  compara¬ 
tive  analysis  of  evaluation  results  of  the  Source  selection 
Evaluation  Board  (SSEB)  (23:5).  The  SSAC  receives  and 
approves  evaluation  standards  developed  by  the  Program 
Office  and  determines  if  it  is  desirable  to  weight  evalua¬ 
tion  criteria  (23:5). 

The  SSEB  is  a  group  of  government  personnel  represen¬ 
ting  various  functional  and  technical  areas  relevant  to  the 
acquisition  (23:5).  The  SSEB  evaluates  proposals  and 
reports  to  the  SSAC.  Evaluation  of  proposals  is  accom¬ 
plished  through  an  in-depth  review  and  evaluation  of  each 
proposal  against  solicitation  requirements,  approved  evalua 
tion  criteria,  and  the  standards  (23:5). 


Fig.  1.  Source  Selection  Organization  [19] 


The  Source  Selection  Plan 


The  Source  Selection  Plan  (SSP)  is  the  key  document 
initiating  a  source  selection.  The  SSP  contains  procedures 
used  to  evaluate  proposals  for  contract  award.  According 
to  APR  70-15,  the  following  is  to  be  contained  in  the  SSP: 

Describe  the  specific  evaluation  criteria 
including,  areas,  items,  and  where  appropriate, 
factors  and  subfactors.  Describe  the  assessment 
criteria  and  how  they  apply  to  the  evaluation. 

Hie  relative  importance  of  all  evaluation 
criteria  will  be  stated  .  .  .  [23:8] 

The  Program  Office  is  responsible,  among  other 
things,  for  establishing  evaluation  criteria  for  SSA 
approval  as  part  of  the  SSP  (23:6).  The  Program  Office 
establishes  the  relative  importance  of  the  evaluation 
criteria  in  the  SSP  in  a  form  for  use  in  the  solicitation 
and  develops  evaluation  standards  for  SSAC  approval  (23:6). 

The  Source  Selection  Plan  Preparation  Guide,  APR 
70-15/ASD  Pamphlet  800-7  dated  June  1985  emphasizes  the 
importance  of  including  R&M  in  the  SSP.  The  guide  states, 
"A  topic  that  must  be  thoroughly  assessed  during  develop¬ 
ment  of  the  SSP  is  Product  Assurance”  (22:1).  As  R&M  falls 
within  the  area  of  Product  Assurance,  the  guide  goes  on  to 
state,  "This  consideration  must  include  reliability,  main¬ 
tainability,  and  producibility  and  any  other  considerations 
needed  to  assure  product  excellence"  (22:1). 


Evaluation  Criteria 


APR  70-15  defines  evaluation  criteria  as,  "The  basis 
for  measuring  each  offeror's  ability  as  expressed  in  its 
proposal,  to  meet  the  government's  needs  as  stated  in  the 
solicitation"  (23:3).  Evaluation  criteria  are  defined  at 
the  time  the  SSP  is  prepared  and  become  a  part  of  the  SSP 
and  must  be  included  in  the  solicitation  (23:8).  Evalua¬ 
tion  criteria  must  be  tailored  to  each  program.  Specific 
criteria  should  be  related  to  characteristics  important  to 
program  success  such  as  maintainability,  producibility, 
supportability,  and  system  effectiveness  (23:9). 

Evaluation  criteria  consist  of  specific  and  assess¬ 
ment  criteria  that  provide  a  matrix  which  identifies  and 
interrelates  what  is  to  be  evaluated  (23:8).  Figure  2 
provides  an  example  of  a  general  format  for  matrix  of  evalu¬ 
ation  criteria. 

Specific  criteria  relate  to  program  characteristics 
and  include  areas  of  evaluation.  Areas  of  evaluation  are 
subdivided  into  items,  factors,  and  at  times,  subfactors. 
The  level  of  subdivision  depends  on  the  complexity  of  the 
area  being  evaluated  (23:9).  Typical  areas  include; 
technical,  logistics,  manufacturing,  operational  utility, 
test,  and  management  (23:9).  Assessment  criteria  relate  to 
the  offeror *3  proposal  and  abilities  and  may  be  ranked  in 
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order  of  relative  importance  (23:9).  Assessment  criteria 
normally  include  the  following  (23:9): 

a.  Soundness  of  technical  approach 

b.  Understanding  of  the  requirement 

c.  Compliance  with  the  requirement 

d.  Past  performance 

e.  Impact  on  the  schedule 


GENERAL  FORMAT  FOR  MATRIX  OF  EVALUATION  CRITERIA 

AREA 

(Technical.  Logistics,  Test,  Management,  etc.) 


NOTES; 

1 .  If  a  factor  is  displayed  graphically  it  must  be  color  coded. 

2.  If  one  factor  for  an  item  is  displayed,  all  factors  for  all  items  within  the  area  must  be  displayed. 


Fig.  2.  Matrix  of  Evaluation  Criteria  [23:9] 


Figure  3  provides  a  visual  representation  of  the  relation¬ 
ship  between  areas,  items,  and  factors.  The  figure  also 
portrays  the  increased  emphasis  being  placed  on  relia¬ 
bility,  maintainability,  and  producibility  by  moving  these 
considerations  from  factors  to  an  item.  As  stated  in 
current  guidance  contained  in  Interim  ASD  Supplement  1 
(Revision  1)  to  AFR  70-15  dated  21  March  1985, 


Coequal  with  technical  performance  as  number  one 
ranking  in  the  technical  area  will  be  relia¬ 
bility  and  maintainability;  producibility  and 
quality  engineering  will  also  rank  close  to  the 
top  of  the  technical  area.  In  addition,  the 
applicable  instructions  to  offerors  must 
instruct  offerors  to  demonstrate  thoroughly  and 
precisely  in  their  proposals  how  reliability, 
maintainability,  and  producibility  are  weighed 
in  the  design  of  the  system  [24:3]. 


The  above  guidance  is  also  contained  in  the  ASD 
Source  Selection  Plan  Preparation  Guide  dated  17  June  1985 
(22:1)  . 


Evaluation  Standards 

According  to  AFR  70-15,  evaluation  standards  esta¬ 
blish  the  minimum  acceptable  level  of  compliance  with  a 
requirement  that  must  be  offered  for  a  proposal  to  be 
considered  acceptable  (23:9).  Standards  may  be  either 
quantitative  or  qualitative  depending  on  the  factor  or 
subfactor  they  address  (23:9).  Standards  are  also  used  for 
measuring  how  well  each  offeror's  approach  meet3  the 
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SOURCE  SELECTION  EVALUATIONS 


Fig.  3.  Source  Selection  Evaluations 


requirements  and  to  determine  when  an  offeror  fails  to  meet 
requirements,  or  exceeds  requirements  (23:9). 

Standards  are  not  included  in  the  SSP  or  the  solicita¬ 
tion  and  are  not  to  be  released  to  any  potential  offeror  or 
personnel  not  directly  involved  in  the  source  selection 
(23:9).  Safeguarding  of  standards  is  required  to  prevent 
offerors  from  obtaining  an  unfair  advantage  through 
receiving  standards  prior  to  submission  of  proposals. 

After  contract  award,  standards  relating  to  a  specific 
contract  are  safeguarded  to  help  prevent  disputes  arising 
from  offerors  who  did  not  receive  the  contract  and  who  may 
seek  to  refute  specific  standards. 

Evaluation  of  proposals  by  the  SSEB  is  conducted  by 
measuring  each  proposal  against  objective  standards  esta¬ 
blished  at  the  lowest  level  of  subdivision  (23:9). 

Proposals  are  not  compared  against  each  other  (23:9). 

Reliability  and  Maintainability  Literature  Review 

DODD  5000.40  defines  reliability  and  maintainability 
as  follows: 


Reliability  -  The  duration  or  probability  of 
failure- free  performance  under  stated  condi¬ 
tions. 

Maintainability  -  The  ability  of  an  item  to  be 
retained  in  or  restored  to  specified  conditions 
when  maintenance  is  performed  by  personnel 
having  specified  skill  levels,  using  prescribed 
procedures  and  resources,  at  each  prescribed 
level  of  maintenance  and  repair  [15:10]. 
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The  general  policy  concerning  R&M  as  stated  in  DODD  5000.40 
is  as  follows: 


Reliability  engineering  shall  focus  on  the 
prevention,  detection  and  correction  of  design 
deficiencies,  weak  parts,  and  workmanship 
defects.  Maintainability  engineering  shall 
reduce  maintenance  and  repair  time,  number  of 
tasks  required  for  each  preventative  and  correc¬ 
tive  maintenance  action,  and  the  need  for 
special  tools  and  test  equipment.  Program  plans 
shall  stress  early  investment  in  R&M  engineering 
in  order  to  avoid  subsequent  costs  and  schedule 
delays  [15:2]. 


This  philosophy  of  early  investment  in  R&M  engi¬ 
neering  must  be  carried  into  the  source  selection  process. 
The  R&M  engineering  policy  states  that,  "The  DOD  components 
shall  define  fundamentals  of  design,  manufacture,  and 
management  which  will  result  in  delivery  of  reliable  and 
maintainable  items  to  the  operational  forces"  (15:2).  DODD 
5000.40  identifies  fundamentals  in  design,  manufacturing, 
and  engineering  tasks  and  tests  (15:2).  The  level  of  R&M 
built  into  a  system  is  to  a  great  extent  dependent  upon  the 
intended  use  of  the  system  and  required  level  of  effi¬ 
ciency.  As  stated  in  DODD  5000.40, 


Tradeoffs  between  performance  and  reliability, 
and  among  required  values  for  system  R&M  para¬ 
meters,  shall  balance  the  design  effort  devoted 
to  operational  effectiveness  with  that  devoted 
to  ownership  cost  reduction  [15:4]. 


As  further  evidence  of  the  need  to  insure  R&M  is  effec¬ 
tively  considered  as  criteria  and  standards  in  source  selec 
tion,  DODO  5000.40  states  that. 


R&M  growth  is  required  during  full  scale  develop¬ 
ment,  concurrent  development  and  production 
(where  concurrency  is  approved) ,  and  during 
initial  deployment.  Predicted  R&M  growth  shall 
be  stated  as  a  series  of  intermediate  mile¬ 
stones,  with  associated  goals  and  thresholds  for 
each  of  these  phases  [15:4]. 


Throughout  the  major  weapon  system  acquisition 
process,  source  selections  are  made.  R&M  ^ust  be  stressed 
during  each  phase  and  within  each  source  selection.  This 
is  supported  in  DODD  5000.40  as  follows: 


Program  review  and  decision  authorities  shall 
address  R&M  achievements  of  the  preceding  phase, 
and  preparations  for  the  following  phase,  at 
each  major  milestone  decision  or  equivalent 
point  in  the  acquisition  process  [15:5]. 


DODD  5000.40  identifies  key  areas  of  R&M  within  each  phase 
of  the  major  weapon  system  acquisition  process  (15:5).  The 
phases  include  mission  analysis,  conceptual,  demonstration 
and  validation,  full  scale  development,  and  production  and 
deployment. 

AFR  800-18  stresses  the  importance  of  considering  R&M 
throughout  the  acquisition  process  as  follows: 


Each  acquisition  and  system  manager  will  assign 
an  R&M  focal  point  and  implement  an  R&M  program. 
The  program  will  support  cost,  schedule,  perfor¬ 
mance  and  supportability  considerations  of  the 
acquisition  and  operational  phases  of  a  system. 


The  definition ,  development,  production,  and 
operation  of  a  system  will  be  guided  by  real¬ 
istic  R&M  requirements  developed  during  the 
conceptual  and  validation  phases.  R&M  qualita 
tive,  quantitative,  and  program  requirements 
will  be  imposed  and  enforced  in  acquisition 
contracts  [18:2]. 


APR  800-18  addresses  the  use  of  R&M  in  the  source  selection 
process: 


The  R&M  programs  proposed  by  contractors  are  key 
factors  to  be  considered  during  source  selec¬ 
tion.  They  are  evaluated  for  their  impact  on 
system  design,  system  effectiveness,  and  life 
cycle  costs  [18:3]. 


As  in  DODD  5000.40,  AFR  800-18  addresses  R&M 
considerations  within  each  phase  of  the  major  weapon  system 
acquisition  process.  AFR  800-18  goes  into  much  greater 
detail  on  these  considerations  than  does  DODD  5000.40.  R&M 
must  be  considered  during  each  phase  of  the  weapon  system 
acquisition  process  and,  concurrently,  within  each  source 
selection. 


III.  MethodolQ' 


Introduction 

This  chapter  will  discuss  the  approach  used  to  study 
current  procedures  used  in  the  selection  of  appropriate 
evaluation  criteria  and  standards  for  R&M  in  source 
selection. 

The  determination  of  appropriate  criteria  and  stan¬ 
dards  for  R&M  is  a  complex  process.  In  order  to  effec¬ 
tively  study  the  process,  experienced  personnel  involved  in 
the  establishment  of  source  selection  criteria  and  stan¬ 
dards  for  R&M  were  interviewed. 

Population 

In  order  to  obtain  a  broad  perspective  on  the  applies 
tion  of  R&M  in  Air  Force  Source  Selections,  personnel  from 
Air  Force  Systems  Command  (AFSC) ,  Aeronautical  Systems 
Division  (ASD)  System  Program  Offices  (SPOs)  and  the  Air 
Force  Acquisition  Logistics  Center  (AFALC)  were  inter¬ 
viewed.  SPOs  at  ASD  are  involved  in  a  multitude  of 
programs  ranging  from  major  weapon  systems  to  subsystems. 
Personnel  involved  in  development  of  R&M  criteria  and  stan¬ 
dards  at  the  major  weapon  system  and  subsystem  level 
comprise  the  research  population. 


Data  Base 


Personnel  interviewed  within  ASD  SPOs  and  AFALC  were 
R&M  Engineers  experienced  in  the  development  of  R&M 
criteria  and  standards.  These  individuals  are  also 
involved  in  implementing  R&M  in  the  source  selection 
process. 

Research  Instrument 

A  questionnaire  was  developed  which  served  as  the 
basis  for  questions  asked  during  interviews.  The  nature  of 
the  research  necessitated  development  of  open-ended 
questions.  Validation  of  the  questionnaire  took  place 
through  completion  of  the  first  two  interviews.  As  a 
result  of  these  interviews,  the  questionnaire  was  reduced 
to  11  questions  from  an  original  14  questions.  The  first 
two  interviews  reflect  the  revised  11  questions  as  do  subse 
quent  interviews.  Insight  into  the  establishment  of  R&M 
criteria  and  standards  was  gained  through  the  personal 
experiences  of  those  interviewed.  The  following  questions 
were  administered  during  the  personal  interviews: 

1.  What  sources  of  reference  are  used  for 
developing  R&M  criteria  and  standards?  If  previous 
contracts  are  used,  what  should  have  or  could  have  been 
used? 


2.  Is  there  a  list  of  criteria  and  standards 
(boiler  plate)  from  which  to  choose  appropriate  criteria  to 


tailor  to  each  system?  what  are  key  areas  that  make  one 
system  unique  from  another? 

3.  What  is  done  to  insure  that  requirements  in  the 
Statement  of  Work  (SOW)  are  stated  in  such  a  way  as  to 
allow  for  the  development  of  evaluation  criteria?  Are  R&M 
personnel  involved  in  RFP  preparation/review  to  help  insure 
proper  statement  of  requirements? 

4.  What  can  be  done  to  improve  stated  requirements 
to  insure  that  evaluation  standards  can  be  developed? 

5.  Are  standards  ranked  in  such  a  way  as  to  give 
proper  weight  to  more  critical  R&M  requirements  in  the  RFP? 

6.  At  what  level  is  R&M  currently  applied  in  source 
selection  (item,  factor,  subfactor)?  How  was  R&M  applied 
prior  to  R&M  2000? 

7.  What  have  you  found  to  be  the  easiest  and  most 
difficult  R&M  areas  to  evaluate?  Why? 

8.  Are  there  any  recurring  problems  in  evaluating 
R&M?  were  contractors  non-responsive  to  any  of  the  R&M 
requirements? 

9.  How  difficult  has  it  been  to  involve  other 
personnel  from  various  functional  areas  in  R&M  evaluation? 

10.  Do  you  believe  R&M  was  properly  addressed  in 
source  selection  prior  to  R&M  2000?  Do  you  believe  R&M 
2000  will  be  effective  in  emphasizing  R&M  in  source 
selection? 


11.  What  recommendations  do  you  have  for  improving 
development  of  criteria  and  standards  for  R&M? 

The  basis  for  establishing  each  of  the  above  listed 
questions  was  as  follows: 

Question  1  was  developed  to  obtain  a  list  of  refer¬ 
ences  for  individuals  involved  in  development  of  criteria 
and  standards.  The  references  are  not  meant  to  be  all 
inclusive,  however,  are  intended  to  provide  a  general  list 
of  available  sources  of  reference. 

Question  2  was  developed  to  determine  if  a  document 
was  available  which  personnel  could  use  as  a  guide  when 
developing  criteria  and  standards.  A  list  of  criteria  and 
standards  would  provide  basic  guidance  as  to  appropriate 
wording  and  areas  to  address.  Appendix  J,  along  with 
providing  a  basic  philosophy  to  follow  in  developing 
criteria  and  standards,  will  also  contain  examples. 

Question  3  addressed  the  important  issue  of  ensuring 
that  SOW  requirements  are  stated  in  such  a  way  that 
criteria  can  be  developed.  SOW  requirements  must  be 
explicit  in  order  to  allow  development  of  criteria  that 
will  be  effective  in  determining  the  contractor's  capa¬ 
bility.  It  is  important  that  R&M  personnel  are  involved 
throughout  the  process  of  RFP  preparation  and  review.  This 
ensures  that  R&M  requirements  are  properly  stated  and  do 
not  conflict  with  other  requirements  in  the  RFP. 


Question  4  seeks  to  determine  ways  to  improve  stated 
requirements  to  allow  development  of  standards.  Standards 
must  be  able  to  judge  particular  capabilities  of  contrac¬ 
tors.  Standards  must  judge  contractors  equally  and  be 
developed  from  clearly  stated  requirements  in  the  SOW. 

Question  5  was  developed  to  determine  if  ranking  of 
standards  takes  place.  Certain  R&M  requirements  are  more 
critical  than  others,  therefore,  a  method  should  exist  for 
placing  greater  weight  on  more  critical  requirements. 

Question  6  was  designed  to  determine  the  extent  of 
implementation  of  current  guidance  which  increases  the 
priority  of  R&M  in  source  selection. 

Question  7  was  developed  to  determine  those  areas 
within  R&M  that  were  easier  and  more  difficult  to  evaluate. 
Answers  to  this  question  provide  areas  that  need  to  be 
stressed  as  criteria  and  standards  and  also  helps  ensure 
personnel  are  prepared  to  evaluate  the  more  difficult 
areas. 

Question  8  also  provides  guidance  as  to  R&M  areas 
which  may  pose  problems  to  evaluators.  Evaluators  must 
attempt  to  resolve  and  prevent  recurring  problems. 

Question  9  evaluates  the  extent  to  which  R&M 
personnel  receive  support  from  various  functional  areas. 

In  order  for  R&M  to  be  effectively  implemented  at  the 


organizational  level,  Command  level,  and  Air  Force  level, 
personnel  within  all  functional  areas  must  be  involved  in 
emphasizing  R&M  throughout  the  acquisition  process. 

Question  10  was  developed  to  determine  the  perspec¬ 
tive  of  R&M  Engineers  as  to  the  impact  of  R&M  2000. 

Question  11  was  designed  to  allow  R&M  Engineers  exper 
ienced  in  the  development  of  criteria  and  standards  to 
provide  personal  recommendations  and  insights  for  improving 
the  process  of  developing  criteria  and  standards. 


Data  Analysis 

Analysis  of  the  interview  results  will  not  involve 
quantitative  procedures,  but  rather  a  qualitative  review 
and  compilation  of  interview  responses.  An  analysis 
consisting  of  an  integration  of  responses  to  the  open-ended 
questions  allows  the  overall  views  of  those  interviewed  to 
be  readily  available.  Individual  responses  to  the  ques¬ 
tions  are  contained  in  the  appendices. 
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IV.  Findings 


Introduction 

This  chapter  provides  the  findings  of  the  research 
based  upon  responses  obtained  in  the  personal  interviews. 
Comprehensive  answers  to  each  of  the  research  questions  are 
contained  in  the  appendices.  Here,  the  responses  will  be 
summarized  and  categorized  according  to  a  key  word  or 
phrase  contained  in  each  question  administered  during  the 
interviews. 

As  expressed  in  the  problem  statement.  Chapter  I, 
guidance  is  dispersed  as  to  methods  of  properly  esta¬ 
blishing  R&M  source  selection  criteria  and  standards.  The 
personal  interviews  provided  first  hand  knowledge  of  the 
process  used  to  establish  R&M  criteria  and  standards.  The 
findings  will  present  concensus  in  certain  areas  as  well  as 
conflicts  and  criticisms  in  others.  Overall,  a  comprehen¬ 
sive  review  of  the  process  is  given  based  on  the  opinions 
of  experienced  R&M  Engineers  from  several  Program  Offices 
at  ASD  and  within  AFALC. 

Appendix  J  utilizes  the  findings  along  with  addi¬ 
tional  information  received  in  the  interviews  such  as  check 
lists  and  examples  of  R&M  criteria  and  standards  to  form  a 
guide  to  the  development  of  R&M  criteria  and  standards. 

The  guide  provides  broad  guidance  not  peculiar  to  a 


specific  system  or  stage  of  the  acquisition  process, 
although  at  times  qualifications  are  made  to  specify  a 
particular  size  program  or  stage  of  acquisition. 

References  Used 

In  developing  criteria  and  standards,  the  basic 
source  of  reference  is  the  Statement  of  Work  (SOW).  The 
SOW  establishes  the  requirements  of  the  RFP.  In  order  to 
evaluate  contractors  based  on  certain  requirements,  the 
requirements  must  be  expressed  in  the  evaluation  criteria 
contained  in  Section  L  of  the  RFP,  Instructions  to  offeror. 

Several  of  those  interviewed  indicated  that  previous 
contracts  were  referenced.  The  extent  to  which  a  previous 
contract  can  be  used  depends  on  the  similarity  among 
systems  being  acquired.  In  some  instances  only  the  format 
from  the  previous  contracts  was  used  whereas  in  others, 
only  minor  changes  were  necessary  in  order  to  use  the 
criteria  and  standards.  Regardless  of  the  degree  of  simil- 
arity  between  systems,  previous  contract  criteria  and 
standards  if  used  as  a  reference  must  always  be  tailored  to 
the  system  being  acquired. 

Several  Military  Standards  and  Handbooks  on  R&M  are 
available.  The  standards  and  Handbooks  provide  detailed 
guidance  on  particular  R&M  tasks  and  requirements.  The 
most  frequently  referenced  document  was  MIL  STD  785B  and 
MIL  STD  781C. 
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The  majority  of  those  interviewed  had  a  vast  amount 


of  experience  in  the  development  of  R&M  criteria  and 
standards.  Those  interviewed  with  less  experience  used  the 
experience  of  others.  On  the  surface,  this  may  seem  a 
subtlety,  however,  it  is  often  under  utilized.  Given  the 
large  number  of  diverse  programs  at  ASD,  a  wealth  of  know¬ 
ledge  exists  which  less  experienced  personnel  should  take 
advantage  of  when  developing  criteria  and  standards. 

List  of  Criteria  and  Standards 

No  approved  list  of  criteria  and  standards  exists 
from  which  to  tailor  to  a  system.  Several  of  those  inter 
viewed  developed  personal  guidelines  based  on  experience 
and  previous  contracts.  One  example  of  a  boiler  plate  type 
of  outline  did  exist  for  use  in  developing  criteria  and 
standards  and  is  contained  in  the  guide  in  Appendix  J. 

An  area  identified  as  making  one  contract  unique  from 
another  was  whether  the  system  was  a  major  system  or 
subsystem.  A  major  system  generally  requires  more  exten¬ 
sive  R&M  criteria  and  standards. 

Whether  a  system  is  mainly  electrical  or  mechanical 
also  makes  a  system  unique.  R&M  requirements  must  be 
tailored  to  account  for  this  difference.  For  example,  R&M 
test  requirements  will  differ  for  an  electrical  system 
versus  a  mechanical  system. 


Stating  Requirements  for  Criteria  Development 

R&M  personnel  must  be  involved  throughout  RFP  prepara 
tion  and  review  to  insure  R&M  requirements  are  properly 
stated. 

To  allow  for  development  of  criteria,  requirements  in 
the  SOW  should  be  stated  in  general  terms  without  detail  to 
the  extent  of  influencing  the  contractor's  design.  Also, 
the  same  person  developing  the  SOW  should  also  develop  the 
criteria.  This  provides  for  consistency  of  thought  and 
allows  for  development  of  criteria  by  an  individual  with 
full  knowledge  and  background  of  the  SOW  requirements. 

Requirements  should  be  stated  in  commercial  terms  to 
allow  contractors  with  less  experience  in  government  work 
to  be  able  to  understand  the  requirements.  This  would  help 
in  development  of  standards  as  contractors  are  less  likely 


to  be  non-responsive  to  certain  RFP  requirements. 

At  times,  R&M  requirements  are  stated  in  both  the 
technical  and  logistics  areas.  However,  R&M  should  only  be 
stated  in  one  of  the  areas,  preferably  technical  to  avoid  a 
conflicting  statement  of  requirements. 

Requirements  should  not  influence  or  restrict  the 
design  process  during  the  conceptual  and  demonstration/ 
validation  phases.  Therefore,  the  particular  phase  in  the 
acquisition  process  should  also  be  considered  when 
developing  SOW  requirements. 


Stating  Requirements  for  Standards  Development 


Standards  are  essentially  used  to  determine  whether 
or  not  the  contractor  has  satisfied  the  requirements  stated 
in  the  evaluation  criteria  in  the  RFP.  Criteria  must  be 
stated  in  such  a  way  that  contractors  submit  justification 
and  data  to  support  their  ability  to  perform  the  various 
requirements.  Contractors  must  not  simply  restate  the  SOW. 
Standards  are  developed  to  judge  the  ability  of  the 
contractor  to  meet  the  criteria. 

As  with  development  of  criteria,  requirements  in  the 
SOW  must  not  be  too  specific.  They  must  be  stated  clearly 
and  unambiguously.  There  is  a  fine  line  between  too 
specific  and  too  general,  however,  the  design  process  must 
not  be  inhibited.  The  same  person  developing  the  SOW 
requirements  and  criteria  should  also  develop  the 
standards. 

A  checklist  type  of  document  would  help  in  the 
development  of  standards.  The  SOW  requirements  could  be 
cross-referenced  with  important  areas  to  consider  in  the 
standards.  Each  acquisition  would  require  tailoring, 
however,  key  areas  to  be  considered  as  standards  could  be 
identified. 

R&M  tasks  must  be  tailored  to  the  specific  system. 

At  times,  tasks  are  incorporated  into  the  RFP  and  not 
tailored.  This  causes  difficulty  in  developing  standards 
to  evaluate  R&M  capability  for  a  specific  system. 
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Ranking  Standards 

An  official  ranking  of  standards  does  not  take  place, 
however,  standards  are  subjectively  ranked.  The  subjective 
ranking  depends  upon  the  evaluator.  Certain  R&M  require¬ 
ments  may  be  considered  more  important  than  others, 
therefore,  the  ability  of  the  contractor  to  demonstrate 
capability  in  the  more  important  areas  will  have  a  greater 
influence  in  determining  compliance  with  the  overall  R&M 
requirement. 

When  R&M  is  considered  as  an  item  in  source  selec¬ 
tion,  factors  and  subfactors  can  be  assigned  a  certain 
weighting.  The  weighting  assigned,  however,  is  subjective. 
A  factor  such  as  R&M  predictions  may  receive  a  certain 
ranking.  Predictions  can  indicate  the  degree  to  which  the 
contractor  understands  R&M  requirements  when  predictions 
are  not  in  the  general  range  of  the  government  estimate. 

Current  Application  of  R&M  in  Source  Selection 

For  the  majority  of  those  Program  Offices  inter¬ 
viewed,  R&M  is  considered  as  the  number  one  item  under  the 
technical  area  in  source  selection.  Where  R&M  has  not 
reached  the  item  level,  priority  has  increased  from  the 
subfactor  to  the  factor  level.  Prior  to  R&M  2000,  the 
majority  of  Program  Offices  considered  R&M  at  the  factor  or 
subfactor  level.  It  appears  that  R&M  2000  has  provided  the 
necessary  impetus  to  allow  R&M  to  receive  the  increased 
priority. 
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in  Evaluating  R&M 


Difficulty 

The  SOW  tasks  such  as  test  plans  and  specifications 
were  identified  as  easier  R&M  requirements  to  evaluate. 
Evaluation  of  data  submitted  was  identified  as  a  difficult 
area  to  evaluate.  At  times ,  contractors  submit  volumes  of 
data.  It  is  up  to  the  evaluator  to  determine  if  the  data 
is  reasonable  or  makes  sense. 

Proposals  submitted  during  the  conceptual  phase  of 
acquisition  are  also  difficult  to  evaluate  as  requirements 
are  vague.  There  is  often  no  past  information  available 
from  which  to  judge  the  reasonableness  of  proposals. 

Future  capability  is  difficult  to  evaluate.  When  a  system 
will  not  be  produced  for  5-10  years ,  it  must  be  determined 
if  the  contractor  will  have  sufficient  capability  in  the 
future.  Many  uncertainties  exist  in  evaluating  this  area. 

Evaluation  of  R&M  during  the  full  scale  development 
and  production  phases  of  acquisition  are  easier.  During 
these  phases,  R&M  tasks  and  requirements  are  fully  identi¬ 
fied;  therefore,  evaluators  know  what  contractors  need  to 
provide  in  proposals  to  adequately  show  R&M  capability. 

Contractor  analysis  of  R&M  includes  identification  of 
critical  R&M  items  in  the  system.  It  is  difficult  to 
dispute  the  selection  of  critical  items  by  the  contractor. 
Also,  in  order  to  evaluate  critical  items,  the  evaluator 


must  be  familiar  with  the  data  submitted  along  with  proce¬ 
dures  used  to  collect  the  data  and  field  maintenance 
procedures.  Contractor  R&M  predictions  also  involve  an 
indepth  evaluation  and  are  often  difficult  to  dispute. 

Maintainability  in  general  is  difficult  to  evaluate 
as  it  is  difficult  to  conceptualize  maintainability  on 
paper.  It  often  takes  hands-on  experience  with  a  system  to 
identify  maintainability  problems.  Evaluators  must  often 
rely  only  on  data  and  designs  submitted  to  determine  the 
maintainability  of  the  system. 

Past  performance  of  a  contractor  on  R&M  requirements 
is  difficult  to  evaluate  as  information  is  often  not  avail¬ 
able  in  the  source  selection  on  R&M  past  performance. 

The  evaluation  of  R&M  budgeted  growth  curves  is 
difficult.  Often  times,  determination  of  start  points  and 
slopes  are  a  matter  of  judgement.  Data  is  not  available  to 
refute  the  contractor's  submission. 

Recurring  Problems  in  Evaluating  R&M 

A  recurring  problem  has  been  a  lack  of  sufficient 
data  submitted  with  proposals.  Also,  at  times  there  is  a 
lack  of  technical  understanding  of  certain  R&M  require¬ 
ments.  This  may  be  indicated  by  a  contractor  being  non- 
responsive  to  an  R&M  requirement.  Non-responsiveness  may 
also  indicate  that  R&M  requirements  or  criteria  were  not 
clearly  stated. 
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Analysis  of  data  is  a  problem  when  a  vast  amount  of 
data  is  submitted  and  limited  time  is  given  to  determine 
the  accuracy  of  the  data. 

Contractors  have  been  reluctant  to  perform  environ¬ 
mental  stress  screening  at  the  parts  level  as  contractors 
claim  the  screening  is  performed  before  receipt  of  the 
parts.  As  a  result,  this  requirement  was  generally  waived 
in  the  past  by  the  government. 

A  negative  attitude  toward  R&M  in  industry  has  caused 
problems.  This  may  have  been  a  reflection  of  government 
emphasis  on  R&M.  Contractors  will  generally  respond  to 
those  areas  where  emphasis  is  placed. 

At  times,  source  selection  management  may  cause 
problems  when  guidance  is  changed  as  to  the  documentation 
that  is  required  during  the  source  selection. 


Involvement  of  Other  Personnel 

It  has  not  been  difficult  to  involve  personnel  from 
various  functional  areas  in  the  evaluation  of  R&M.  The 
degree  to  which  personnel  outside  the  R&M  function  become 
involved  is  dependent  on  the  size  and  complexity  of  the 
particular  program. 

With  the  current  emphasis  on  R&M,  other  workload  on 
R&M  engineers  has  increased  due  in  part  to  the  number  of 
questions  raised  that  require  elaborate  replies.  This 
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makes  it  more  difficult  to  involve  experienced  R&M 
engineers  in  the  actual  sou-  _e  selection.  Therefore, 
proper  statement  of  criteria  and  standards  is  becoming 
increasingly  important  due  to  the  lack  of  experienced  R&M 
personnel  in  source  selection. 

R&M  2000  Impact 

The  main  concensus  among  those  interviewed  was  that 
R&M  requirements  for  a  weapon  system  were  properly 
addressed  in  the  past.  However,  R&M  did  not  receive  the 
required  priority  in  source  selection.  Prior  to  R&M  2000, 
it  was  up  to  the  discretion  of  source  selection  management 
to  determine  the  priority  of  R&M.  At  times,  this 
discretion  was  used  to  loosen  the  interpretation  of  R&M 
requirements. 

There  had  been  a  grass  roots  push  to  increase  the 
priority  of  R&M  prior  to  R&M  2000.  R&M  2000  provided  the 
necessary  upper  level  management  attention  and  Air  Force¬ 
wide  attention  to  make  R&M  the  prime  focus  of  attention. 

Those  interviewed  believe  that  R&M  2000  will  be  effec 
tive  in  emphasizing  R&M  in  source  selection.  Upper  level 
management,  however,  must  continue  to  support  and  emphasize 


Recommendations  for  Improving  Development  of  Criteria  and 
Standards 

Several  recommendations  for  improving  development  of 
criteria  and  standards  for  R&M  were  provided  by  those  inter 
viewed.  The  recommendations  are  listed  below: 

1.  The  ultimate  R&M  objectives  for  the  program  must 
be  kept  in  mind.  The  criteria  and  standards  serve  as  the 
means  of  obtaining  the  objectives. 

2.  Source  selection  standards  should  be  written 
prior  to  release  of  the  RFP.  When  standards  are  not 
written  at  the  same  time  as  the  SOW  and  evaluation 
criteria,  different  personnel  may  be  tasked  to  write  the 
standards.  This  causes  inconsistency  of  thought  and  also  a 
loss  of  knowledge  as  to  system  peculiarities  that  were 
known  by  the  individual  who  developed  the  SOW  and  criteria. 

3.  Personnel  should  not  hesitate  to  seek  the  advice 
of  an  experienced  R&M  engineer  when  tasked  with  developing 
criteria  and  standards. 

4.  R&M  could  be  more  effectively  evaluated  if  opera¬ 
tional  terms  could  be  directly  correlated  with  contractual 
terms.  It  is  often  difficult  to  determine  if  operational 
requirements  are  satisfied  even  when  all  contractual 
requirements  have  been  fulfilled.  In  order  to  translate  an 
operational  requirement  into  contractual  terms,  several 
subsystem  MTBFs  and  other  measures  must  be  combined. 
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5.  Additional  time  should  be  built  into  acquisition 
milestones  to  allow  for  proper  development  of  the  RFP  SOW 
for  R&M.  Proper  development  of  criteria  and  standards 
hinges  on  a  well  written  SOW. 

6.  R&M  Engineers  should  ensure  that  Program  Managers 
become  familiar  with  R&M  as  it  relates  to  their  particular 
program.  This  will  help  ensure  proper  emphasis  is  placed 
on  R&M  throughout  the  various  phases  of  the  program. 

7.  Additional  guidance  in  the  form  of  checklists  or 
examples  would  improve  the  development  of  criteria  and 
standards.  R&M  Engineers  should  develop  personal  lists  of 
standards  and  detailed  questions  for  use  in  future 
programs. 

8.  Increase  the  number  of  R&M  Engineers.  Current 
emphasis  on  R&M  through  R&M  2000  causes  increased  workload 
on  R&M  Engineers  in  the  form  of  additional  documentation 
requirements,  development  of  policy,  and  replies  to  higher 
level  inquiries.  Additional  manpower  is  needed  to  handle 
the  increased  workload  to  allow  sufficient  time  for  develop 
ment  of  criteria  and  standards. 


V.  Conclusions  and  Recommendations 


Introduction 

This  chapter  will  restate  the  problem  statement  in 
order  to  determine  the  extent  to  which  the  identified 
problem  was  solved  through  this  research  effort.  The 
conclusion  will  summarize  the  approach  used  and  findings  of 
the  research  which  led  to  solution  of  the  problem.  This 
chapter  will  also  provide  recommendations  for  further 
research. 

Problem  Statement 

Proper  evaluation  of  R&M  in  the  source  selection 
process  is  imperative  to  insure  the  most  reliable  and  main¬ 
tainable  weapon  systems  are  acquired,  however,  guidance  is 
dispersed  as  to  methods  of  properly  establishing  R&M  source 
selection  evaluation  criteria  and  standards. 

A  study  of  current  guidance  for  applying  R&M  source 
selection  evaluation  criteria  and  standards  is  needed  to 
provide  personnel  with  direction  as  to  effective  evaluation 
of  R&M  in  source  selection. 

Conclusions 

Through  personal  interviews  with  R&M  engineers  exper¬ 
ienced  in  the  development  of  R&M  evaluation  criteria  and 
standards,  current  guidance  and  procedures  for  establishing 
R&M  criteria  and  standards  was  studied. 
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The  interviews  provided  valuable  personal  experience 
and  data  which  served  as  the  basis  for  establishment  of  a 
guide  for  development  of  source  selection  evaluation 
criteria  and  standards  for  R&M.  The  guide  is  contained  in 
Appendix  J. 

As  identified  in  the  problem  statement,  guidance  is 
dispersed  as  to  methods  of  properly  establishing  R&M  source 
selection  evaluation  criteria  and  standards.  Appendix  J 
provides  a  concise  compilation  of  guidance  and  procedures 
to  use  when  developing  criteria  and  standards  for  R&M.  The 
guide  provides  personnel  involved  in  the  development  of 
criteria  and  standards  with  a  systematic  process  and  impor¬ 
tant  areas  that  must  be  taken  into  consideration  to  develop 
effective  R&M  criteria  and  standards. 

With  the  current  emphasis  being  placed  on  the  impor¬ 
tance  of  R&M,  in  particular,  the  increased  priority  of  R&M 
in  the  source  selection  process,  this  thesis  has  provided 
research  results  that  help  implement  the  current  emphasis. 
In  order  to  acquire  weapon  systems  that  are  more  reliable 
and  maintainable,  R&M  must  be  addressed  in  source  selec¬ 
tion.  For  R&M  to  be  effectively  addressed,  evaluation 
criteria  and  standards  must  be  properly  developed.  This 
thesis  serves  to  ensure  proper  development  of  R&M  source 
selection  evaluation  criteria  and  standards. 


Recommendations  for  Further  Research 

This  research  has  identified  areas  which  should  be 
pursued  in  further  research  efforts.  The  following 
recommendations  for  further  research  are  provided: 

1.  Research  should  be  conducted  to  determine  an 
effective  way  of  translating  operational  R&M  requirements 
into  contractual  R&M  requirements  and  vice  versa.  It  is 
often  difficult  to  determine  if  operational  requirements 
are  satisfied  even  when  all  contractual  requirements  are 
met.  To  translate  an  operational  requirement  into  a 
contractual  requirement,  several  contractual  measures  such 
as  MTBF  and  MTBM  must  be  combined  to  satisfy  an  operational 
situation.  An  effective  method  of  translating  operational 
requirements  into  contractual  requirements  would  allow 
personnel  to  more  clearly  specify  R&M  requirements  in  the 
SOW,  and  as  source  selection  criteria  and  standards. 

2.  Research  should  also  be  conducted  to  determine  a 
ranking  structure  or  weighting  of  R&M  within  source  selec¬ 
tion.  With  R&M  elevated  in  source  selection  to  the  number 
one  item  under  the  technical  area,  the  R&M  item  will  have 
various  requirements  as  factors  and  subfactors.  Certain 
R&M  requirements  are  more  critical  to  the  success  of  a 
system  than  others.  More  critical  R&M  requirements  should 
be  identified  and  a  weighting  system  developed  to  ensure 
that  these  requirements  receive  greater  consideration  in 
determining  the  most  reliable  and  maintainable  system. 
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Appendix  A:  Personal  Interview 


Name:  Robert  Armstrong 

Organization:  Airlift  and  trainer,  ASD/AFES 
Duty  Title:  R&M  Engineer 

Experience:  Development  of  R&M  criteria  and  standards. 

Advisor  for  the  Advanced  Rescue  Helicopter. 

Item  Captain  for  the  Next  Generation  Trainer. 

Question  1 — Previous  contracts  were  used  for 
reference  as  well  as  personal  experience  of  others  in 
development  of  criteria  and  standards.  Mr.  Armstrong  had 
no  prior  experience  in  this  area  and  found  little  guidance 
available. 

Question  2 — Sample  formats  existed  which  were 
tailored  to  the  system.  No  unique  areas  were  identified. 

Question  3 — Avoid  ambiguous  statements  in  the  SOW. 
Criteria  should  be  written  when  the  SOW  is  written.  R&M 
personnel  are  involved  throughout  RFP  preparation  and 
review. 

Question  4 — Requirements  should  be  stated  in  such  a 
way  that  contractors  who  do  not  deal  with  the  government 
often  can  understand  the  requirements.  For  instance,  for 
the  Next  Generation  Trainer,  Cessna  Corporation  had  trouble 
with  certain  requirements  as  their  business  is  mainly 
commercial  rather  than  military. 

Question  5 — Subjective  ranking  of  standards  takes 
place.  There  is  no  official  ranking.  It  would  be  hard  to 
assign  weighting  to  the  standards. 
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Question  6 — R&M  is  considered  as  an  item  under  the 
technical  area  as  well  as  an  item  under  the  logistics  area. 
This  was  being  done  prior  to  R&M  2000. 

Question  7 — The  easiest  areas  are  the  SOW  tasks  such 
as  test  plans  and  schedules.  It  is  difficult  to  evaluate 
R&M  predictions  and  growth  curves.  Contractors  did  not  use 
past  data  to  estimate  the  growth  curves  and  the  government 
had  little  knowledge  of  what  the  slope  of  the  growth  curves 
should  be. 

Question  8 — A  recurring  problem  is  analysis  of  the 
data  submitted.  At  times,  volumes  of  data  can  be  submitted 
with  little  time  to  evaluate. 

Question  9 — It  has  not  been  difficult  to  involve 
other  personnel  in  R&M  evaluation. 

Question  10 — R&M  was  not  properly  addressed  prior  to 
R&M  2000.  There  was  not  enough  emphasis  on  R&M.  There  was 
little  interest  in  the  details  of  a  good  R&M  program.  It 
was  addressed  because  it  had  to  be.  R&M  2000  will  be 
effective  if  upper  management  continues  the  emphasis. 

Question  11 — Development  of  criteria  and  standards 
would  improve  if  a  checklist  was  available  which  indicated 
important  areas  to  consider.  Personnel  should  also  learn 
what  to  expect  from  a  proposal.  This  would  help  avoid 
asking  for  too  little  or  too  much  information. 


Appendix  B:  Personal  Interview 
Name:  Wallace  Detert 

Organization:  Tactical  Systems,  ASD/TAES 

Duty  Title:  R&M  Engineer 

Experience:  Development  of  R&M  criteria  and  standards  for 
the  C-141,  C-130 ,  C-5 ,  A-10,  F-15,  P-16,  ATF, 
and  a  number  of  subsystems. 

Question  1 — Previous  contracts  for  similar  systems 
are  often  a  source  of  reference.  Criteria  and  standards 
must  be  tailored  to  the  particular  system.  Various  R&M 
handbooks  are  available  detailing  R&M  specifications  and 
tasks.  MIL  STD  785  and  MIL  STD  781  are  also  used  as 
sources  of  reference. 

Question  2 — There  is  no  list  of  criteria  available. 
Key  areas  on  a  system  are  the  fasteners.  Mr.  Detert  has 
observed  numerous  hours  being  spent  replacing  defective 
fasteners. 

Question  3 — The  SOW  must  be  clearly  delineated  and 
succinct.  Criteria  should  not  be  too  detailed  as  to  tie 
the  contractors  hands.  The  SOW  must  consider  the  parti¬ 
cular  phase  of  acquisition  as  the  SOW  requirements  should 
not  influence  or  restrict  the  design  process  during  concep¬ 
tual  and  demonstration/validation  phases.  R&M  personnel 
are  involved  with  RFP  preparation  and  review. 

Question  4 — Experienced  personnel  must  be  involved  in 
the  development  of  the  SOW.  A  checklist  approach  should 
not  be  used  as  each  acquisition  is  unique.  The  person 


developing  the  SOW  should  also  develop  the  standards. 
Standards  must  not  be  too  detailed  as  to  inhibit  evalua¬ 
tions.  Contractors  must  be  free  to  submit  innovative 
approaches  to  satisfy  government  requirements.  Standards 
must  be  developed  in  such  a  way  that  innovative  approaches 
can  be  evaluated. 

Question  5 — Generally  weighting  of  standards  is  not 
accomplished.  Certain  areas  of  R&M  receive  subjective 
ranking  which  impact  on  the  overall  evaluation  of  R&M. 

Question  6 — R&M  is  currently  the  number  one  item 
under  the  technical  area.  R&M  was  previously  considered  as 
a  factor. 

Question  7 — Evaluating  the  contractor's  analysis  of 
R&M  is  the  most  difficult.  Analysis  deals  with  the  identi¬ 
fication  by  the  contractor  of  critical  R&M  items  within  a 
system.  Analysis  must  be  done  correctly  to  insure  proper 
design  of  a  system.  In  order  to  evaluate  the  contractor's 
analysis,  one  must  be  familiar  with  data  collected  as  well 
as  procedures  used  to  collect  data.  Field  maintenance 
procedures  must  also  be  known. 

Question  8 — A  recurring  problem  has  been  a  negative 
attitude  toward  R&M  in  industry. 

Question  9 — It  has  not  been  difficult  to  involve 
other  personnel  in  R&M  evaluation.  Recently,  personnel 
have  become  too  involved  in  the  sense  that  personnel  who 
are  not  well  versed  in  R&M  are  asking  questions  due  to  the 


emphasis  of  R&M  2000.  Questions  from  upper  levels  necessi¬ 
tate  elaborate  replies  which  have  tended  to  increase  the 
workload  among  an  undermanned  R&M  Engineering  force. 

Question  10 — R&M  was  properly  addressed  prior  to  R&M 
2000.  Many  things  have  been  learned  over  the  years.  The 
C-5  aircraft  included  full  R&M  requirements  at  every  level. 
R&M  was  addressed  more  from  mission  reliability  rather 
than  logistics  reliability.  Today,  mission  and  logistics 
reliability  receive  attention.  R&M  2000  provides  upper 
level  management  support  of  R&M  which  is  necessary.  R&M 
2000  will  be  effective  in  emphasizing  R&M. 

Question  11 — Experienced  personnel  must  be  involved 
in  development  of  criteria  and  standards.  An  inexperienced 
person  developing  criteria  and  standards  should  receive 
guidance  from  experienced  personnel. 


Appendix  C:  Personal  Interview 
Name:  1st  Lt  William  Liberti 

Organization:  Reconnaissance  and  Electronic  Warfare 
Systems,  ASD/RWEX 
Duty  Title:  R&M  Engineer 

Experience:  Development  of  criteria  and  standards  for  SEEK 

RAM  program. 

Question  1 — 1st  Lt  Liberti  was  given  little  direction 
when  tasked  with  developing  R&M  criteria  and  standards. 
Sources  of  reference  included  the  SOW,  specifications, 
previous  contracts,  MIL  HDBK  217D,  MIL  STD  756,  and 
experience  of  other  personnel. 

Question  2 — A  boiler  plate  type  of  outline  was  used 
for  development  of  the  R&M  criteria.  Portions  of  the 
boiler  plate  are  included  in  the  R&M  guide,  Appendix  J. 

The  R&M  Maintenance  concept  can  make  one  system  unique  from 
another.  For  the  SEEK  RAM,  the  contractor  has  the  option 
of  two  or  three  levels  of  maintenance. 

Question  3 — The  SOW  and  criteria  should  be  developed 
at  the  same  time.  This  helps  provide  consistency.  Communi¬ 
cation  between  R&M  and  Logistics  should  also  take  place  to 
avoid  duplication  of  requirements.  R&M  personnel  are 
involved  throughout  RFP  preparation  and  review. 

Question  4--Requi.rements  must  not  be  too  specific  as 
to  constrain  the  contractor.  Requirements  should  be 
general  enough  to  allow  the  contractor  to  develop  an 


effective  R&M  plan.  There  is  a  fine  line  between  too 
specific  and  too  general. 

Question  5 — A  subjective  weighting  takes  place. 
Certain  factors  and  subfactors  within  the  R&M  item  can  be 
considered  more  important  than  others.  R&M  predictions 
often  provide  a  good  comparison  among  contractors  as  a 
contractor  will  be  readily  identified  if  predictions  are 
not  within  the  general  range  of  the  government  estimate. 

Question  6 — R&M  is  considered  as  two  separate  items 
under  the  technical  area.  Prior  to  R&M  2000,  R&M  was  as  a 
factor  under  the  performance  item. 

Question  7 — Maintainability  is  difficult  to  evaluate 
as  it  is  difficult  to  conceptualize  on  paper.  It  is  diffi¬ 
cult  to  determine  if  the  design  is  the  most  efficient  for 
maintainability.  Under  Reliability  Task  102,  monitoring  of 
subcontractors,  it  is  difficult  to  evaluate  if  the  contrac¬ 
tor's  system  will  be  effective.  Reliability  critical  items 
are  also  difficult  to  evaluate.  It  is  difficult  to  dispute 
what  the  contractor  considers  to  be  critical  items. 

Budgeted  growth  curves  are  also  difficult  to  evaluate  as  it 
is  hard  to  determine  if  the  entry  point  and  slope  are 
accurate.  It  is  up  to  to  the  evaluator's  judgement  to 
determine. 

Question  8 — Contractors  are  reluctant  to  perform 
environmental  stress  screening  at  the  parts  level  as  they 


claim  the  screening  is  done  before  they  receive  the  part. 
The  requirement  was  generally  waived  by  the  government. 

Question  9 — It  has  not  been  difficult  to  involve 
other  personnel  in  R&M  evaluation. 

Question  10 — R&M  was  not  properly  addressed  in  the 
sense  that  it  was  not  taken  seriously  although  it  was  a 
requirement.  R&M  2000  is  effective  in  emphasizing  the 
importance  of  R&M  to  those  outside  of  the  R&M  field. 

Question  11 — Personnel  involved  in  development  of  r&m 
criteria  and  standards  must  keep  in  mind  the  ultimate  R&M 
objectives  of  the  program.  The  criteria  and  standards 
serve  as  the  means  of  obtaining  the  objective. 


Appendix  D:  Personal  Interview 
Name:  William  Lucka 

Organization:  Product  Assurance,  ASD/ENSI 
Duty  Title:  Division  Chief 

Experience:  Developing  R&M  criteria  and  standards. 

Directing  R&M  policy  to  Program  Offices. 

Question  1 — The  major  source  of  reference  is  the  SOW 
Along  with  the  SOW,  Mr.  Lucka  advised  familiarity  with  MIL 
STD  785  as  it  provides  guidance  on  the  rationale  for  using 
particular  R&M  tasks.  At  times,  previous  contracts  are 
used  as  a  guide,  however,  fresh  thinking  must  always  be 
used.  Criteria  and  standards  should  be  developed  with  the 
frame  of  mind  of  what  the  government  ultimately  wants  to 
achieve. 

Question  2 — There  is  no  list  of  criteria  and  stan¬ 
dards  from  which  to  choose.  Criteria  and  standards  are 
unique  in  respect  to  major  weapon  systems  versus 
subsystems.  There  are  similarities,  however,  major  weapon 
system  criteria  and  standards  are  more  extensive. 

Question  3 — To  insure  development  of  criteria,  the 
government  should  avoid  having  two  R&M  requirements;  one 
under  Technical  and  the  other  under  Logistics.  At  times. 
Logistics  personnel  attempt  to  write  R&M  requirements 
without  knowing  the  design  process  and  do  not  coordinate 
with  R&M  engineers.  This  results  in  requirements  being 
stated  referencing  specific  types  of  equipment  rather  than 


stating  requirements  in  terms  of  performance.  Logistics 
coordination  is  necessary,  however.  Product  Assurance 
should  have  the  main  responsibility  for  developing  R&M 
requirements  with  R&M  under  the  Technical  area. 

Question  4 — Personnel  tend  to  be  too  specific  and 
will  tell  the  contractors  how  to  design.  Requirements 
should  be  stated  clearly,  unambiguously,  not  subject  to 
interpretations,  and  not  so  specific  as  to  tie  contractors' 
hands  in  telling  how  to  design.  An  example  of  misleading 
requirements  is  to  specify  that  the  system  will  be 
supported  at  two  levels  of  maintenance;  flight  line  and 
depot,  when  in  fact,  intermediate  level  of  maintenance  will 
also  be  provided. 

Question  5 — Standards  are  not  specifically  ranked, 
however,  subjective  ranking  takes  place.  Certain  standards 
are  weighed  stronger  in  determining  whether  or  not  the 
contractor  has  mec  the  overall  R&M  requirements. 

Question  6 — R&M  is  being  applied  in  source  selection 
as  the  number  one  item  under  the  technical  area.  Mr.  Lucka 
is  not  aware  of  R&M  consideration  prior  to  R&M  2000. 

Question  7 — N/A.  Not  involved  in  recent  source 
selection. 

Question  8 — A  major  problem  is  a  lack  of  sufficient 
data  submitted  by  contractors.  Sufficient  back  up  material 
is  not  submitted  to  substantiate  R&M  capability. 


Question  9 — It  has  not  been  difficult  to  involve 
personnel  in  source  selections,  however,  it  is  often  diffi¬ 
cult  after  source  selection  to  involve  personnel. 

Question  10 — R&M  was  properly  addressed  prior  to  R&M 
2000.  There  had  been  a  push  to  increase  R&M  priority.  R&M 
2000  provided  the  attention  necessary  to  increase  the 
priority  of  R&M.  R&M  2000  will  be  effective  in  emphasizing 
R&M. 

Question  11 — R&M  may  be  better  evaluated  if  opera¬ 
tional  terms  could  be  directly  correlated  with  contractual 
terms.  It  is  often  difficult  to  determine  if  operational 
requirements  are  satisfied  even  when  all  contractual 
requirements  are  fulfilled.  An  example  of  the  difficulty 
would  be  an  operational  statement  such  as  95%  probability 
of  success  of  a  one-and-a-half  hour  mission.  In  order  to 
translate  this  requirement  into  contractual  terms,  several 
subsystem  MTBF's  and  other  measures  must  be  combined. 


Appendix  E:  Personal  Interview 
Name:  Clay  Nieman 

Organization:  Aeronautical  Equipment,  ASD/AEES,  INU 

Program  Office. 

Duty  Title:  R&M  Engineer 

Experience:  Involved  in  a  number  of  system  acquisitions 

developing  R&M  criteria  and  standards. 

Programs  include:  Locust  missile,  DADS,  INU, 
among  others. 

Question  1 — A  major  factor  considered  when  devel¬ 
oping  R&M  criteria  and  standards  is  the  requirements  in  the 
SOW.  Mr.  Nieman  creates  a  checklist  of  key  R&M  areas  in 
the  SOW  with  paragraph  references.  This  insures  that  all 
necessary  areas  are  addressed  in  the  criteria.  Criteria 
must  indicate  types  of  information  that  the  government 
expects  to  be  submitted  in  proposals.  Contractors  should 
not  simply  state  that  requirements  can  be  met,  they  must 
state  detailed  plans  and  procedures  for  accomplishing  the 
task. 

Along  with  the  referenced  paragraphs,  Mr.  Nieman 
makes  notes  of  what  he  expects  to  see  in  the  proposals  to 
insure  the  areas  are  adequately  covered  by  the  contractor 
in  the  proposal.  This  constitutes  the  standards  used  in 
the  source  selection. 

A  second  factor  considered  is  the  phase  of  acquisi¬ 
tion.  Concept  exploration  has  limited  R&M  tasking  as 
system  design  i3  in  preliminary  stages,  therefore,  criteria 
and  standards  will  be  less  comprehensive.  Demonstration/ 


Validation  has  greater  tasking.  Tasking  peaks  at  Pull 
Scale  Development  as  the  design  becomes  finalized  and 
decreases  in  production. 

Mr.  Nieman  listed  the  following  sources  of  reference 
that,  as  a  minimum,  personnel  involved  in  development  of 
criteria  and  standards  should  be  familiar  with: 


DOD  Directive  5000.40 — Reliability  and 

Maintainability 

AFR  800-18 — Air  Force  Reliability  and 
Maintainability  Program 

MIL  STD  785B — Reliability  Program  for  Systems  and 

Equipment  Development  and  Production. 
MIL  STD  470A — Maintainability  Program  for  Systems 
and  Equipment. 

MIL  STD  756B — Reliability  Modeling  and  Prediction 
MIL  HDBK  217D — Reliability 
MIL  STD  781C — Reliability  Testing 
MIL  STD  1629A — FMECA 
AS  4613 — Derating  Criteria 
MIL  HDBK  472 — Maintainability  Prediction 
MIL  STD  471A--Maintainability/Built-In-Test 
Demonstration 
MIL  STD  2165— Testability 
MIL  HDBK  338— Reliability 

MIL  HDBK  189 — Reliability  Growth  Management 
MIL  STD  810 — Environmental  Test 
MIL  STD  883B — Electronic  Component  Testing 
MIL  STD  1635 — Reliability  Growth  Standard 
MIL  HDBK  251 — Thermal  Analysis 


Mr.  Nieman  stressed  that  criteria  and  standards  must  be 
tailored  to  each  acquisition. 

Question  2 — Mr.  Nieman  did  not  indicate  any  avail¬ 
able  list  of  criteria  and  standards  or  key  areas  making  a 
system  unique.  Once  again,  he  stated  that  criteria  and 
standards  must  be  tailored.  Mr.  Nieman  has  compiled  a 
comprehensive  list  of  questions  to  be  asked  and  areas  to  be 


addressed  in  R&M  criteria  and  standards.  Portions  of  the 
list  will  be  contained  in  Appendix  J. 

Question  3 — To  insure  requirements  in  the  SOW  are 
stated  in  a  way  to  allow  development  of  criteria,  the  same 
person  developing  the  SOW  should  also  develop  the  criteria. 
This  cannot  always  be  accomplished  due  to  PCSs  and 
personnel  shortages.  R&M  personnel  are  involved  in  RPP 
preparation  and  review. 

Question  4 — To  improve  stated  requirements,  adequate 
information  must  be  contained  in  the  instructions  to 
offeror  section  to  insure  the  proposal  provides  more  than  a 
restatement  of  the  SOW. 

Question  5 — Reliability  and  Maintainability  stan¬ 
dards  are  equal  in  priority.  Ranking  is  done  at  the  area 
level. 

Question  6 — Mr.  Nieman  indicated  that  R&M  is 
receiving  attention  at  the  factor  level  under  the  item 
technical  specialties/product  assurance.  Previous  to  R&M 
2000,  technical  specialties/product  assurance  was  a  factor 
and  R&M  was  a  subfactor.  This  indicates  that  R&M  has  moved 
up  in  priority. 

Question  7 — Future  capability  has  been  the  most 
difficult  R&M  area  to  evaluate.  When  a  contractor  is 
proposing  a  system  which  will  not  be  built  for  5-10  years, 
a  lot  of  judgement  is  required.  Many  uncertainties  exist. 


Question  8 — Mr.  Nieman  has  not  run  into  recurring 
problems.  If  a  contractor  is  non-responsive  to  an  R&M 
requirement,  it  generally  means  that  the  contractor  does 
not  have  the  capability  or  requirements  are  not  stated 
properly. 

Question  9 — It  has  not  been  difficult  to  involve 
other  personnel  in  R&M  evaluation.  Generally,  the 
following  personnel  are  involved;  systems  engineering. 
Logistics,  manufacturing,  and  cost  analysis. 

Question  10 — R&M  was  properly  addressed  prior  to  R&M 
2000,  however,  it  did  not  receive  the  attention  or 
priority.  R&M  2000  will  improve  R&M,  however,  Mr.  Nieman 
sees  a  manpower  constraint  in  the  future  with  a  shortage  of 
R&M  engineers. 

Question  11 — Recommendations: 

1.  Increase  the  number  of  R&M  engineers. 

2.  Provide  more  time  for  development  of  the  SOW  to 
insure  proper  criteria  and  standards  can  be  developed. 

3.  Program  Managers  should  be  more  thoroughly 


briefed  on  R&M. 


Appendix  P:  Personal  Interview 
Name:  Richard  Papenbrock 

Organization:  Reconnaissance  and  Electronic  Warfare, 

ASD/RWEX 

Duty  Title:  Lead  Engineer,  Product  Assurance 
Experience:  Developing  criteria  and  standards  for  R&M. 

Responsible  for  48  programs  ranging  from 
conceptual  phase  to  production  phase. 

The  eleven  questions  were  not  administered  to  Mr. 
Papenbrock  as  he  provided  an  overview  of  the  emphasis  being 
placed  on  R&M  within  his  organization.  R&M  is  considered 
as  an  item  under  the  technical  area  in  source  selection. 

R&M  is  considered  as  one  item  or  broken  out  into  two  items; 
reliability  and  maintainability. 

Upper  level  management  attention  is  placed  on  R&M 
through  the  policy  that  Dr.  Halpin,  ASD/PA  will  be  involved 
in  all  source  selections.  Mr.  Papenbrock  has  developed  an 
operating  instruction,  01  800-8,  Product  Assurance  Program, 
which  integrates  R&M,  Logistics,  quality,  and  related  func¬ 
tional  areas  to  more  effectively  influence  the  design  and 
development  process.  This  10  is  being  used  within  many 
SPOs  at  ASD. 

Prior  to  R&M  2000,  Mr.  Papenbrock' s  office  was  accom¬ 
plishing  required  R&M  tasks  along  with  initiating  emphasis 
on  R&M.  Although  R&M  2000  has  provided  the  attention 
needed  at  higher  levels,  it  has  also  increased  workload  in 
respect  to  reporting  of  R&M  and  involvement  in  various 


Appendix  G:  Personal  Interview 


Name:  Robert  M.  Read 
Organization:  AFALC/PTRR 

Duty  Title:  R&M  Engineer 

Experience:  Developing  criteria  and  standards  for  the 

Automated  Remote  Tracking  System.  Evaluated 
R&M  for  the  Advanced  Tactical  Fighter  in 
Conceptual  Phase. 

Question  1 — The  SOW  and  specifications  are  the  main 
references  used  in  developing  evaluation  criteria  contained 
in  the  Instructions  to  Offeror  (ITO).  In  developing  stan¬ 
dards  ,  only  those  requirements  in  the  ITO  can  be  evaluated 
against  specific  standards.  MIL  STD  785B  and  MIL  STD  470 
are  basic  references  that  personnel  should  be  familiar  with 
when  developing  R&M  criteria  and  standards. 

Question  2 — A  list  of  criteria  and  standards  did  not 
exist,  however,  certain  criteria  were  used  after  tailoring 
them  for  the  specific  system.  A  consideration  which  makes 
one  system  unique  from  another  is  whether  the  system  is 
more  mechanical  or  electrical.  SOW  requirements  must  be 
written  to  accommodate  this  difference.  R&M  test  plan 
requirements,  for  example,  will  be  different  for  a  mechan¬ 
ical  system  versus  an  electrical  system.  Each  system  acqui 
sition  is  unique,  therefore,  criteria  and  standards  must  be 
written  to  reflect  the  differences  among  systems. 

Question  3 — Requirements  should  be  explicit  and 
clearly  identify  to  the  contractor  what  is  to  be  achieved. 
The  contractor  should  not  be  told  how  to  design.  R&M 


personnel  are  involved  throughout  RPP  preparation  and 
review  to  insure  that  R&M  requirements  do  not  conflict  with 
other  requirements  in  the  RFP. 

Question  4 — R&M  tasks  must  be  tailored  to  the 
specific  system.  At  times ,  tasks  are  taken  from  the  mili¬ 
tary  standards  and  not  tailored.  This  causes  difficulty  in 
developing  standards  to  evaluate  R&M  capability  for  a 
specific  system. 

Question  5 — Mr.  Read  has  not  observed  ranking  of 
standards,  however,  ranking  should  take  place  as  certain 
R&M  requirements  are  more  critical  than  others. 

Question  6 — R&M  is  currently  the  number  one  item 
under  the  technical  area.  R&M  has  been  moving  up  in 
priority  over  the  past  few  years.  Prior  to  R&M  2000,  R&M 
was  considered  as  a  factor  or  subfactor. 

Question  7 — Evaluation  of  R&M  predictions  is  an 
easier  area  to  evaluate.  Although  calculations  are 
required  to  verify  proposed  predictions,  there  are  specific 
areas  to  check. 

Comparability  analysis  in  the  conceptual  phase  of 
acquisition  is  difficult.  The  analysis  is  subjective  and 
deals  with  determining  future  capability  and  technology 
compared  to  that  existing  today. 


Question  8 — A  recurring  problem  is  a  lack  of  suffi¬ 
cient  data  submitted  with  proposals.  Contractors  do  not 
support  stated  R&M  capability. 

Question  9 — Prior  to  the  current  emphasis  on  R&M,  it 
was  difficult  to  involve  other  personnel.  Today  it  is  not 
difficult. 

Question  10 — R&M  was  being  properly  addressed, 
however,  R&M  2000  provided  the  necessary  upper  level  manage 
ment  support  to  increase  the  priority  of  R&M.  R&M  2000  is 
being  effective  in  emphasizing  R&M  in  source  selection. 

Question  11 — Personnel  must  insure  that  requirements 
in  the  SOW  and  specifications  are  clearly  written  as  these 
serve  as  the  basis  for  development  of  criteria  and 
standards. 


Appendix  Hs  Personal  Interview 


Name :  1st  Lt  Michael  Secen 

Organization:  AFALC/PTRR 
Duty  Title:  R&M  Engineer 

Experience:  Developing  R&M  criteria  and  standards  for  the 

Integrated  Electronic  Warfare  System,  EF-111A 
upgrade,  and  others. 

Question  1 — The  SOW  is  the  main  source  of  reference. 

A  major  reference  is  also  MIL  STD  785B.  A  reliability 
design  checklist  developed  by  the  U.S.  Navy  is  also 
utilized. 

Question  2 — A  list  of  criteria  and  standards  is  not 
available.  1st  Lt  Secen  is  not  aware  of  areas  that  make 
one  system  unique  from  another. 

Question  3 — R&M  personnel  are  involved  in  RFP 
preparation.  Requirements  should  be  written  in  general 
terms.  AFALC  interfaces  with  Product  Assurance  to 
incorporate  logistics  concerns  into  R&M  requirements. 

Question  4 — To  improve  development  of  standards, 
standards  should  be  written  when  the  RFP  is  prepared. 
Difficulty  arises  in  developing  standards  when  the  same 
person  developing  the  RFP  SOW  is  not  involved  in  developing 
standards.  Having  the  same  individual  develop  standards 
provides  consistency  in  development.  An  individual's 
experience  on  a  particular  system  will  not  be  lost. 
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Question  5 — Ranking  of  standards  takes  place  through 
subjective  ranking  by  the  evaluators.  A  formalized  ranking 
is  not  established. 

Question  6 — R&M  is  being  pushed  as  the  number  one 
item  under  the  technical  area.  R&M  was  previously  consi¬ 
dered  at  the  factor  or  subfactor  level. 

Question  7 — It  is  difficult  to  evaluate  proposals  in 
the  conceptual  phase  of  acquisition  as  requirements  are 
vague.  Contractor  past  performance  on  R&M  requirements  is 
also  difficult  to  evaluate  as  little  information  is 
provided  in  source  selection  on  past  performance.  Evalua¬ 
tion  is  less  difficult  during  the  full  scale  development 
and  production  phases  as  specific  tasks  and  requirements 
are  identified  and  evaluators  know  what  the  contractor 
needs  to  provide  to  show  R&M  capability. 

Question  8 — Problems  arise  with  source  selection 
management  when  guidance  is  changed  in  regards  to  required 
documentation.  Management  is  not  always  familiar  with  R&M 
requirements  in  source  selection. 

Question  9 — Personnel  have  generally  been  involved  in 
their  own  areas.  R&M  has  received  adequate  support. 

Question  10 — R&M  was  emphasized  prior  to  R&M  2000, 
however,  R&M  2000  provides  the  needed  emphasis  from  upper 
level  management.  R&M  2000  will  be  effective  in  empha¬ 
sizing  R&M  in  source  selection. 
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Question  11 — Standards  should  be  written  before 


release  of  the  RFP.  Source  selection  management  should 


Appendix  Is  Personal  Interview 
Name:  Gene  Wolanski 

Organization:  Product  Assurance,  ASD/ENSI 

Duty  Title:  R&M  Engineer 

Experience:  Involved  in  development  of  R&M  criteria  and 

standards  for  subsystems  such  as  modular 
automatic  test  equipment,  ground  power 
generator,  towbar,  and  others. 

Question  1 — Criteria  and  standards  are  developed  from 
the  SOW.  Format  from  previous  contracts  is  generally  used, 
however,  criteria  and  standards  must  be  tailored  for  each 
acquisition. 

Question  2 — There  is  no  list  of  criteria  and 
standards.  Certain  criteria  and  standards  are  taken  from 
previous  contracts  and  tailored.  Management  generally 
provides  guidance  as  to  the  appropriateness  of  standards. 

Question  3 — Personnel  developing  the  SOW  must  not 
provide  too  much  detail  in  the  requirements  and  thereby 
influence  the  design  of  the  system.  R&M  personnel  are 
involved  in  RFP  preparation  to  help  insure  properly  stated 
requirements. 

Question  4 — A  format  to  follow  in  developing 
standards  would  be  useful.  Requirements  could  be  cross- 
referenced  to  the  standards  format  to  insure  all  areas  are 
covered.  Specific  standards,  however,  cannot  be  developed 
until  the  SOW  and  criteria  are  developed. 

Question  5 — Standards  have  received  equal  ranking. 


Question  6 — R&M  is  being  applied  as  an  item  under  the 
technical  area.  R&M  was  previously  treated  as  a  factor. 

Question  7 — The  R&M  plan  has  been  the  easiest  area  to 
evaluate.  Evaluating  data  submitted  is  the  most  difficult 
area  as  you  must  determine  if  the  data  is  reasonable  or 
makes  sense. 

Question  8 — A  recurring  problem  involves  a  lack  of 
sufficient  data  submitted  with  proposals.  There  is  also, 
at  times,  a  lack  of  technical  understanding  by  the 
contractor. 

Question  9 — Other  personnel  have  been  involved  in  R&M 
evaluation.  The  degree  to  which  personnel  are  involved 
depends  on  the  particular  source  selection  (size  of  the 
program) . 

Question  10 — Prior  to  R&M  2000,  the  degree  to  which 
R&M  was  emphasized  in  source  selection  was  dependent  upon 
the  source  selection  management.  At  times,  program  pres¬ 
sure  caused  R&M  requirements  to  be  loosened.  An  example  of 
this  would  be  a  decision  to  verify  R&M  on  paper  rather  than 
through  demonstration  due  to  pressure  to  decrease  program 
costs.  R&M  2000  will  be  effective  in  emphasizing  R&M  as 
higher  level  attention  is  placed  on  R&M. 

Question  11 — More  guidance  in  the  form  of  checklists 
or  examples  would  improve  the  development  of  criteria  and 


standards. 


Appendix  J:  Guide  for  Development  of  Source  Selection 
Evaluation  Criteria  and  Standards  for 
Reliability  and~MaintainabilTty 


This  guide  was  established  to  aid  personnel  in  the 
development  of  source  selection  evaluation  criteria  and 
standards  for  reliability  and  maintainability. 


GUIDE  FOR  DEVELOPMENT  OF  SOURCE  SELECTION 


EVALUATION  CRITERIA  AND  STANDARDS  FOR 
RELIABILITY  AND  MAINTAINABILITY 


Captain  Daniel  E.  Harnen,  Jr. 


Air  Force  Institute  of  Technology 
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GUIDE  FOR  DEVELOPMENT  OF  SOURCE  SELECTION 
EVALUATION  CRITERIA  AND  STANDARDS  FOR 
RELIABILITY  AND  MAINTAINABILITY 

Introduction 

Reliability  and  Maintainability  (R&M)  issues  have 
become  the  prime  focus  of  attention  within  the  Air  Force  in 
the  development  and  acquisition  of  major  weapon  systems.  A 
weapon  system  must  be  able  to  perform  with  consistent  relia 
bility  and  be  designed  for  efficient  and  effective  maintain 
ability. 

Hie  source  selection  process  is  the  avenue  through 
which  major  weapon  systems  are  brought  into  the  Air  Force 
inventory,  proper  evaluation  R&M  in  the  source  selection 
process  is  imperative  to  insure  the  most  reliable  and  main¬ 
tainable  weapon  systems  are  acquired. 

This  guide  provides  general  procedures  to  follow  when 
developing  source  selection  evaluation  criteria  and  stan¬ 
dards  for  R&M.  The  guide  is  not  intended  as  a  detailed 
step-by-step  procedure  for  development  of  evaluation 
criteria  and  standards  for  a  particular  weapon  system  or 
phase  of  the  acquisition  process.  Each  acquisition  is 
unique ,  therefore,  general  guidance  is  provided  which  may 
be  applied  to  various  acquisitions. 


A  number  of  examples  of  evaluation  criteria  and 
standards  for  R&M  are  provided  in  the  guide.  Although  the 
examples  may  not  specifically  apply  to  the  system  being 
acquired  by  the  users  of  this  guide,  they  do  illustrate 
effective  ways  to  state  various  R&M  requirements.  The 
examples  should  be  tailored  to  the  particular  system  being 
acquired.  Also  contained  in  the  guide  is  a  list  of 
questions  that  evaluators  may  use  to  support  and  clarify 
various  R&M  standards.  The  questions  relate  to  the 
Reliability  Program  Plan,  Maintainability  Program  Plan, 
Built-InTest  (BIT) ,  and  Reliability  Testing.  Finally,  the 
guide  identified  R&M  areas  that  are  easier  and  more 
difficult  to  evaluate. 

Evaluation  Criteria  Defined 
AFR  70-15  defines  evaluation  criteria  as  the  basis 
for  measuring  each  offeror's  ability  as  expressed  in  its 
proposal,  to  meet  the  government's  needs  as  stated  in  the 
solicitation.  Evaluation  criteria  must  be  included  in  the 
solicitation  and  must  be  tailored  to  each  program. 

Evaluation  criteria  consist  of  specific  and  assess¬ 
ment  criteria  that  provide  a  matrix  which  identifies  and 
interrelates  what  is  to  be  evaluated.  Figure  1  provides  an 
example  of  a  general  format  for  matrix  of  evaluation 
criteria. 
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Specific  criteria  relate  to  program  characteristics 
and  include  areas  of  evaluation.  Areas  of  evaluation  are 
subdivided  into  items,  factors,  and  at  times,  subfactors. 
The  level  of  subdivision  depends  on  the  complexity  of  the 
area  being  evaluated.  Typical  areas  include;  technical, 
logistics,  manufacturing,  operational  utility,  test,  and 
management.  Within  the  technical  area,  for  instance,  R&M 
is  considered  as  the  number  one  item  to  be  evaluated. 
Assessment  criteria  relate  to  the  offeror's  proposal  and 
abilities  and  may  be  ranked  in  order  of  relative 
importance.  Assessment  criteria  normally  include  the 
followings 

1.  Soundness  of  technical  approach 

2.  Understanding  of  the  requirement 

3.  Compliance  with  the  requirement 

4.  Past  performance 

5.  Impact  on  the  schedule. 

Figure  2  provides  a  visual  representation  of  the 
relationship  between  areas,  items,  and  factors.  The  figure 
also  portrays  the  increased  emphasis  being  placed  on 
reliability,  maintainability,  and  producibility  by  moving 
these  considerations  from  factors  to  an  item.  Criteria 
must  instruct  offerors  to  demonstrate  thoroughly  and 
precisely  in  their  proposals  how  reliability, 
maintainability,  and  producibility  are  weighed  in  the 
design  of  the  system. 


GENERAL  FORMAT  FOR  MATRIX  OF  EVALUATION  CRITERIA 
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NOTES: 

1 .  If  a  factor  is  displayed  graphically  it  must  be  color  coded. 

2.  If  one  factor  for  an  item  is  displayed,  all  factors  for  all  items  within  the  area  must  be  displayed. 


Fig.  1.  Matrix  of  Evaluation  Criteria 


SOURCE  SELECTION  EVALUATIONS 


Source  Selection  Evaluations 


Evaluation  Standards  Defined 
According  to  AFR  70-15,  evaluation  standards  esta¬ 
blish  the  minimum  acceptable  level  of  compliance  with  a 
requirement  that  must  be  offered  for  a  proposal  to  be  consi 
dered  acceptable.  Standards  may  be  either  quantitative  or 
qualitative  depending  on  the  factor  or  subfactor  they 
address.  Standards  are  also  used  for  measuring  how  well 
each  offeror's  approach  meets  the  requirements  and  to 
determine  when  an  offeror  fails  to  meet  requirements,  or 
exceeds  requirements. 

Standards  are  not  included  in  the  solicitation  and 
are  not  to  be  released  to  any  potential  offeror  or 
personnel  not  directly  involved  in  the  source  selection. 
Safeguarding  of  standards  is  required  to  prevent  offerors 
from  obtaining  an  unfair  advantage  through  receiving 
standards  prior  to  submission  of  proposals.  After  contract 
award,  standards  relating  to  a  specific  contract  are  safe¬ 
guarded  to  help  prevent  disputes  arising  from  offerors  who 
did  not  receive  the  contract  and  who  may  seek  to  refute 
specific  standards. 

References  Used  When  Developing  Criteria  and  Standards 
In  developing  criteria  and  standards,  the  basic 
source  of  reference  is  the  Statement  of  Work  (SOW).  The 
SOW  establishes  the  requirements  of  the  Request  for 
Proposal  (RFP ) .  In  order  to  evaluate  contractors  based  on 


80 


certain  requirements ,  the  requirements  must  be  expressed  in 
the  evaluation  criteria  and  contained  in  Section  L  of  the 
RFP,  Instructions  to  Offeror  (ITO). 

Previous  contracts  may  be  used  as  a  reference, 
depending  on  the  similarity  among  systems  being  acquired. 

In  some  instances  only  the  format  from  the  previous 
contracts  may  be  used,  whereas  in  others,  only  minor 
changes  may  be  necessary  in  order  to  use  the  criteria  and 
standards.  Regardless  of  the  degree  of  similarity  between 
systems,  previous  contract  criteria  and  standards  if  used 
as  a  reference  must  always  be  tailored  to  the  system  being 
acquired. 

The  experience  of  others  may  also  be  used  as  a 
reference.  On  the  surface,  this  may  seem  a  subtlety, 
however,  it  is  often  under  utilized.  Given  the  large 
number  of  diverse  programs  within  the  Air  Force,  a  wealth 
of  knowledge  exists  which  less  experienced  personnel  should 
take  advantage  of  when  developing  criteria  and  standards. 

Several  Military  Standards  and  Handbooks  on  R&M  are 
available.  The  Standards  and  Handbooks  provide  detailed 
guidance  on  particular  R&M  tasks  and  requirements.  As  a 
minimum,  personnel  should  be  familiar  with  the  following 
re ferences: 

DOD  Directive  5000.40 — Reliability  and 

Maintainability 

AFR  800-18 — Air  Force  Reliability  and 
Maintainability  Program 


MIL  STD  785B — Reliability  Program  for  Systems  and 

Equipment  Development  and  Production. 
MIL  STD  470A — Maintainability  Program  for  Systems 
and  Equipment. 

MIL  STD  756B — Reliability  Modeling  and  Prediction 
MIL  STD  781C — Reliability  Testing 
MIL  STD  1629A — FMECA 

MIL  STD  471A — Maintainability/Built-In-Test 
Demonstration 
MIL  STD  2165— Testability 
MIL  STD  810 — Environmental  Test 
MIL  STD  883B — Electronic  Component  Testing 
MIL  STD  1635 — Reliability  Growth  Standard 
AS  4613 — Derating  Criteria 
MIL  HDBK  217D— Reliability 
MIL  HDBK  472 — Maintainability  Prediction 
MIL  HDBK  338 — Reliability 

MIL  HDBK  189 — Reliability  Growth  Management 
MIL  HDBK  251 — Thermal  Analysis 


Stating  Requirements  for  Criteria  Development 
R&M  personnel  must  be  involved  throughout  RFP  prepara¬ 
tion  and  review  to  insure  R&M  requirements  are  properly 


stated. 


To  allow  for  development  of  criteria,  requirements  in 
the  SOW  should  be  stated  in  general  terms  without  detail  to 
the  extent  of  influencing  the  contractor's  design.  Also, 
the  same  person  developing  the  SOW  should  also  develop  the 


criteria  and  standards.  This  provides  for  consistency  of 


thought  and  allows  for  development  of  criteria  and  stan¬ 
dards  by  an  individual  with  full  knowledge  and  background 
of  the  SOW  requirements. 


Requirements  should  be  stated  in  commercial  terms  to 


allow  contractors  with  less  experience  in  government  work 
to  be  able  to  understand  the  requirements.  This  would  help 


in  development  of  standards  as  contractors  are  less  likely 
to  be  non-responsive  to  certain  RFP  requirements. 


At  times ,  R&M  requirements  are  stated  in  both  the 
technical  and  logistics  areas.  However,  R&M  should  only  be 
stated  in  one  of  the  areas,  preferably  technical  to  avoid  a 
conflicting  statement  of  requirements. 

Requirements  should  not  influence  or  restrict  the 
options  of  the  designer  during  the  conceptual  and 
demonstration/validation  phases.  Therefore,  the  particular 
phase  in  the  acquisition  process  should  also  be  considered 
when  developing  SOW  requirements. 

Stating  Requirements  for  Standards  Development 
Standards  are  essentially  used  to  determine  whether 
or  not  the  contractor  has  satisfied  the  requirements  stated 
in  the  evaluation  criteria  in  the  RFP.  Criteria  must  be 
stated  in  such  a  way  that  contractors  submit  justification 
and  data  to  support  their  ability  to  perform  the  various 
requirements.  Contractors  must  not  simply  restate  the  SOW. 
Standards  are  developed  to  judge  the  ability  of  the 
contractor  to  meet  the  criteria. 

As  with  development  of  criteria,  requirements  in  the 
SOW  must  not  be  too  specific.  They  must  be  stated  clearly 
and  unambiguously.  There  is  a  fine  line  between  too 
specific  and  too  general,  however,  the  design  process  must 
not  be  inhibited. 


R&M  tasks  must  be  tailored  to  the  specific  system. 
At  times,  tasks  are  incorporated  into  the  RFP  and  not 
tailored.  This  causes  difficulty  in  developing  standards 
to  evaluate  R&M  capability  for  a  specific  system. 


Development  of  Evaluation  Criteria  for  R&M 


Properly  stated  evaluation  criteria  are  one  of  the 
most  important  elements  of  the  source  selection  process 
because  only  those  areas  stated  as  evaluation  criteria  in 
the  Request  for  Proposal  (RFP)  can  be  evaluated  in  the 
source  selection.  The  evaluation  criteria  are  contained  in 
the  Instructions  to  Offeror  (ITO)  section  of  the  RFP  and 
are  intended  to  inform  the  offeror  of  areas  that  will  be 
evaluated  during  source  selection.  Improperly  stated  or 
inadequate  evaluation  criteria,  therefore,  can  seriously 
affect  the  source  selection  process.  In  order  to  insure 
that  the  contractor  with  the  most  reliable  and  maintainable 
weapon  system  is  chosen  in  source  selection  given  cost 
considerations;  evaluation  criteria  majt  be  properly 
stated. 


Proper  statements  of  evaluation  criteria  will  include 
all  critical  R&M  requirements  from  the  Statement  of  Work 
(SOW)  and  specifications.  Evaluation  criteria  should  not 
simply  be  a  restatement  of  all  SOW  and  specification 
requirements.  It  is  up  to  the  individual  developing 
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criteria  to  choose  critical  R&M  requirements.  When  stating 
criteria,  personnel  must  avoid  language  which  restricts  the 
contractor's  freedom  to  design  the  most  reliable  and  main¬ 
tainable  system. 

Development  of  criteria  should  be  considered  as  a 
systematic  process  which  begins  with  familiarity  of  the  SOW 
and  specifications.  When  developing  criteria,  personnel 
should  keep  in  mind  the  overall  objectives  of  the  weapon 
system  R&M  and  develop  criteria  to  meet  these  objectives. 
Offerors  must  be  instructed  to  demonstrate  how  each  R&M 
task  will  be  accomplished.  Offerors  must  also  be  required 
to  submit  data  to  support  proposed  R&M  capability.  Receipt 
of  adequate  data  to  support  R&M  capability  cannot  be  over¬ 
emphasized.  For  example,  at  time  offerors  simply  state 
that  R&M  requirements  in  the  SOW  will  be  met,  however,  they 
do  not  provide  supporting  data  to  show  the  capability  to 
meet  the  requirements. 

The  particular  phase  in  the  acquisition  process  must 
also  be  considered  when  developing  evaluation  criteria.  In 
the  conceptual  phase  of  acquisition,  detailed  R&M  needs  are 
usually  impractical  to  define,  especially  below  the  system 
level.  However,  based  on  threat  analysis,  force  planning 
and  budgeting  information ,  and  R&M  shortcomings  of  current 
systems,  top  level  R&M  needs  can  be  defined.  Evaluation 
criteria,  however,  will  not  be  extensive  in  the  conceptual 
phase.  R&M  requirements  increase  during  the  demonstration/ 


validation  phase  and  generally  peak  at  full  scale  develop¬ 
ment  as  the  system  design  is  completed.  Evaluation 
criteria  in  the  full  scale  development  phase  must 
adequately  reflect  the  increased  R&M  requirements  in  the 
SOW.  In  the  production  phase,  R&M  requirements  slightly 
decrease,  however,  new  tasks  may  be  added  due  to  changes  in 
design.  Therefore,  evaluation  criteria  must  be  written  to 
reflect  the  particular  phase  of  acquisition. 

To  illustrate  the  relationship  between  the  phase  of 
acquisition  and  degree  of  R&M  tasking.  Tables  I  and  II, 
Application  Matrix  for  Reliability  and  Application  Matrix 
for  Maintainability,  provide  personnel  involved  in  RFP 
preparation  with  guidance  as  to  reliability  and  maintain¬ 
ability  tasks  which  should  be  included  during  various 
stages  of  acquisition.  The  particular  tasks  selected  will 
depend  on  the  system  being  acquired  and  phase  of  acquisi¬ 
tion.  As  can  be  seen,  few  requirements  are  applicable 
during  concept  exploration  while  at  full  scale  development, 
generally  all  tasks  are  applicable.  R&M  tasking  decreases 
during  production. 

The  size  of  the  particular  program  will  impact  devel¬ 
opment  of  evaluation  criteria.  Generally,  for  a  major 
weapon  system  acquisition  versus  a  subsystem  acquisition, 
criteria  will  be  much  more  elaborate  due  to  ths  greater 
amount  of  R&M  requirements  in  the  SOW  and  specifications. 
For  a  subsystem  acquisition,  evaluation  criteria  will 
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Table  I 


TASK 

TITLE 

TASK 

TYPE 

PROGRAM  PHASE 

CONCEPT 

VALID 

FSfiD 

PROD 

101 

HELtABILlTY  KKOGHAH  PLAN 

MGT 

s 

s 

G 

G 

102 

MONITOR/ CONTROL  Ok  SUBCONTRACTORS 

AND  SUPPLIERS 

MGT 

s 

S 

G 

G 

10J 

PROGRAM  REVIEWS 

MOT 

s 

s(2) 

0(2) 

0(2) 

ION 

FAILURE  REPORTING,  ANALYSIS,  AND 
CORRECTIVE  ACTION  SYSTEM  (FRACAS) 

ENG 

NA 

S 

G 

C 

105 

FAILURE  REVIEW  BOARD  (FRB) 

HOT 

NA 

s(2) 

G 

G 

201 

RELIABILITY  MODELING 

ENG 

S 

S(2> 

0(2) 

GC(2) 

202 

RELIABILITY  ALLOCATIONS 

ACC 

s 

G 

G 

GC 

203 

RELIABILITY  PREDICTIONS 

ACC 

S 

S(2) 

G(2) 

GC(2) 

20N 

FAILURE  MODES,  EFFECTS,  AND 
CRITICALITY  ANALYSIS  (FHECA) 

ENG 

s 

S 

( 1 ) (2) 

0 

(1X2) 

GC 

(0(2) 

205 

SHEAR  CIRCUIT  ANALYSIS  (SCA) 

ENG 

NA 

NA 

(Ml) 

0C(1) 

206 

ELECTRONIC  PARTS/CIRCUITS 

TOLERANCE  ANALYSIS 

ENG 

NA 

NA 

0 

GC 

207 

PANTS  PROGRAM 

ENG 

s 

s(2><3> 

0(2) 

0(2) 

208 

RELIABILITY  CRITICAL  ITEMS 

MCT 

SO) 

3(1) 

G 

G 

209 

EFFECTS  OF  FUNCTIONAL  TESTING, 
STORAGE,  HANDLING,  PACEAGINC, 
TRANSPORTATION,  AND  MAINTENANCE 

ENG 

|9 

s(l> 

G 

GC 

301 

ENVIRONMENTAL  STRESS  SCREENING 
(ESS) 

ENG 

B 

S 

0 

302 

RELIABILITY  DEVELOPMENT/CROWTH 
TESTING 

ENG 

SI  2) 

G(2) 

B 

303 

RELIABILITY  QUALIFICATION  TEST 
(ROT)  PROGRAM 

ACC 

NA 

‘  3(2) 

0(2) 

0(2) 

30N 

PRODUCTION  RELIABILITY  ACCEPTANCE 
ACCEPTANCE  TEST  (PRAT)  PROGRAM 

ACC 

NA 

_ 

NA 

S 

0(2X3) 

COM  DgFTNTTTflIIS 


usa  n«: 


PBOCIUH  PtUfifi 


ACC  -  RELIABILITY  ACCOUNTING 
ENG  -  RELIABILITY  ENGINEERING 
HOT  .  MANAGEMENT 


S  -  SELECTIVELY  APPLICABLE 

C  -  GENERALLY  APPLICABLE 

GC  -  GENERALLY  APPLICABLE  TO  DESIGN 
CHANGES  ONLY 


NA  -  NOT  APPLICABLE 

(1)  -  REQUIRES  CONSIDERABLE  INTERPRETATION 

OP  INTENT  TO  BE  COST  EFFECTIVE 

(2)  -  MIL-STD-785  IS  NOT  THE  PRIMARY 

IMPLEMENTATION  REQUIREMENT.  OTHER 
HIL-STDS  OR  STATEMENT  OF  WORK 
REQUIREMENTS  MUST  BE  INCLUDED  TO 
DEFINE  THE  REQUIREMENTS. 


Table  II 


Maintainability  Application  Matrix 


TASK  TITLE 

TASK 

TYPE 

CON¬ 

CEPT 

VALID 

PROGRAM  PHASE 

FSO  PROD 

OPERAT  SYSTEM 
DEV  (MOOS) 

101 

Maintainability  Program 

Plan 

MGT 

N/A 

6(3) 

G 

6(3) (1} 

6(1) 

102 

Monitor/Control  of  Sub* 
contractors  and  Vendors 

MGT 

N/A 

S 

6 

6 

S 

103 

Program  Rev levs 

MGT 

S 

6(3) 

6 

6 

s 

104 

Data  Collection, 

Analysis  and  Corrective 
Action  System 

ENG 

N/A 

S 

6 

G 

s 

201 

Maintainability  Modeling 

ENG 

S 

S<4) 

G 

C 

N/A 

202 

Maintainability 

Allocations 

ACC 

S 

S(4) 

3 

C 

S(4| 

203 

Maintainability 

Predictions 

ACC 

N/A 

S(2) 

6(2) 

C 

S(2) 

204 

Failure  Modes  and 

Effects  Analysis  (FMEAJ 
Maintainability  Information 

ENG 

N/A 

S(2) 

(3) (4) 

6(1) 

(2) 

C(l) 

(2) 

S(2) 

205 

Maintainability  Analysis 

ENG 

S(3) 

3(3) 

30) 

C(D 

S 

206 

Maintainability  Oeslgn 
Criteria 

ENG 

N/A 

S(3) 

G 

C 

s 

207 

Preparation  of  Inputs  to 
Detailed  Maintenance  Plan 
and  Logistics  Support 
Analysis  (ISA) 

ACC 

N/A 

S(2) 

(3) 

6(2) 

C(2) 

s 

301 

Maintainability 

Demonstration  (MO) 

ACC 

N/A 

S(2) 

6(2) 

C(2) 

S(2) 

coot  DEFINITIONS 
S  -  Selectively  applicable 
G  -  Generally  Applicable 

C  -  Generally  Applicable  to  design  changes  only 

N/A  -  Not  applicable 

ACC  •  Maintainability  Accounting 

ENG  -  Maintainability  Engineering 

MGT  -  Management 

(1)  Requires  considerable  interpretation  of  intent  to  be  cost  effective. 

(2)  MlL-STD-470  Is  not  the  primary  Implementation  document.  Other  MIL-STDS  or  StatoMnt  of  More 
requirements  must  be  included  to  define  or  rescind  the  requirements.  For  example  MIL-STD-471 
must  be  Imposed  to  describe  maintainability  demonstration  details  and  methods. 

(3)  Appropriate  for  those  task  elements  suitable  to  definition  during  phase. 

(4)  Depends  on  physical  complexity  of  the  system  unit  being  procured.  Its  packaging  and  Its  overall 
maintenance  policy. 
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generally  be  less  involved  as  fewer  requirements  will  be 
contained  in  the  SOW  and  specifications.  The  following  two 
examples  illustrate  fairly  elaborate  R&M  criteria  which  may 
be  tailored  to  a  major  weapon  system  acquisition: 

Elaborate  Criteria 
Example  1: 

Reliability  and  Maintainability.  Describe  your 
approach  for  assuring  that  the  specified  reliability  and 
maintainability  (R&M)  requirements  will  be  met.  Show  how 
you  will  allocate  reliability  and  identify  reliability 
critical  items,  for  example,  the  radar  transmitter  [insert 
critical  items  peculiar  to  the  system  being  acquired].  For 
all  subsystems,  present  preliminary  R&M  allocations, 
models,  and  predictions  down  to  the  Line  Replaceable  Unit 
(LRU)  level  [Insert  level  for  given  acquisition]  giving 
series  and  mission  mean-time-between-failures  as  defined  in 
the  System  Specification,  and  mean  and  maximum  maintenance 
times.  Identify  sources  of  data  including  demonstrated 
reliability  for  LRUs  that  are  production  derivative 
hardware.  Provide  sample  calculations  for  mission  relia¬ 
bility  and  describe  the  impact  of  and  sensitivity  to  the 
acceptable  performance  levels  defined  in  the  System  Specifi¬ 
cation.  Describe  all  redundancy  and  system  monitoring, 
failure  detection/isolation,  and  built-in-test  (BIT/BITE) 
capabilities,  your  Failure  Reporting,  Analysis  and 
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Corrective  Action  System  (FRACAS) ,  subcontractor/vendor 
controls  and  monitoring ,  identification  and  control  of 
critical  items,  and  environmental  stress  screening  program. 

Example  2: 

Reliability /Maintainability.  The  offeror  shall 
provide  R&M  discussion  and  data,  as  outlined  below,  as  part 
of  the  technical  proposal. 

1.  Preliminary  R&M  program  plans,  which  itemize  and 
discuss  each  task  which  will  be  performed,  shall  be 
submitted  with  the  technical  proposal.  These  plans  shall 
also  include: 

a.  An  organizational  chart  and  explanation  of 
the  management  structure  and  emphasis  that  insure 
accomplishment  of  R&M  tasks. 

b.  A  discussion  of  the  R&M  engineering  role 
relative  to  design  decisions,  program  visibility  and 
contribution  to  the  overall  program. 

2.  Discuss  in  detail  the  design  features  and 
techniques  utilized  to  achieve  compliance  with  R&M 
requirements. 

3.  Provide  preliminary  R&M  predictions  with  detailed 
discussions  of: 

a.  The  methods  and  conditions  used  in  the 
predictions. 

b.  The  testing  to  be  performed. 


c.  The  data  items  to  be  provided. 

d.  A  discussion  of  high  risk  areas  and  their 
program  impacts  and  potential  solutions. 

e.  The  failure  reporting,  analysis,  and  correc¬ 
tive  action  system  used. 

f.  R&M  concepts. 

g.  Analyses  to  be  performed  and  their  potential 

benefits. 

h.  The  preliminary  reliability  predictions  shall 
indicate  the  expected  laboratory  and  field  system  and 
subsystem  reliability  indices  in  terms  of  Mean  Time  Between 
Failure  ( MTBF )  and  Mean  Flying  Time  Between  Maintenance 
(MTBM)  (Inherent)  and  (Total)  and  Mean  time  Between  Removal 
(MTBR)  (which  relates  to  MTBM  (Total))  as  defined  in  AFR 
800-18,  respectively.  The  preliminary  maintainability 
predictions  shall  indicate  the  expected  LRU  and  SRU  main¬ 
tainability  indices  in  manhours  per  flying  hour  and 
Mean-Time-To-Repair  (MTTR)  at  each  level. 

4.  Discuss  in  detail  the  design  features  and  techni¬ 
ques  used  to  achieve  Built-In-Test  (BIT)  confidence  require 
rnents.  Include  a  preliminary  BIT  confidence  prediction. 

5.  Automatic  Test  Equipment  being  proposed  by  the 
offeror  to  support  his  proposed  system  shall  be  discussed 
in  detail,  including: 

a.  Fault  isolation  capability. 


b.  The  specific  level  of  indenture  within  the 


LRU  being  tested. 

6.  Discuss  any  other  support  equipment  and/or  Govern¬ 
ment  Furnished  Equipment  (GFE) . 

Boiler  Plate  Criteria 

The  following  example  may  be  tailored  to  a  subsystem 
acquisition.  The  criteria  are  in  a  "boiler  plate"  format 
and  require  only  the  applicable  paragraph  from  The  SOW. 

Example : 

Reliability  Program.  The  offeror  shall  document  a 
plan  of  the  proposed  reliability  program  including  Growth 
Curves.  The  reliability  program  plan  shall  identify, 
describe,  and  tie  together  those  reliability  program  compon¬ 
ents  described  in  paragraph  101.2  of  Task  101  of  MIL  STD 
785B  as  tailored  to  the  particular  needs  of  this  program  by 
the  Statement  of  Work  Paragraph  _ . 

Maintainability  Program.  The  offeror  shall  docu¬ 
ment  a  plan  of  the  proposed  maintainability  program.  The 
maintainability  program  shall  identify,  describe,  and  tie 
together  those  maintainability  program  components  described 
in  paragraph  101.2  of  Task  101  of  MIL  STD  470A  as  tailored 
to  the  particular  needs  of  this  program  by  the  Statement  of 
Work  paragraph  _ . 


Development  of  Source  Selection  Standards  for  R&M 
Only  those  requirements  stated  vrithin  the  evaluation 
criteria  can  be  used  to  evaluate  offerors  in  source 
selection.  Therefore ,  source  selection  standards  must  be 
developed  direct-tv  from  the  evaluation  criteria.  Standards 
are  established  to  judge  whether  or  not  the  offeror  has  met 
the  requirements  stated  in  the  evaluation  criteria.  In 
order  to  accomplish  this,  standards  are  tied  to  specific 
requirements  in  the  SOW  and  specification  from  which  the 
evaluation  criteria  were  developed. 

The  following  basic  format  and  categories  are  used  in 
several  examples  shown  in  this  guide  and  may  be  used  by 
personnel  developing  standards: 

AREA:  [Indicate  down  to  the 

ITEM:  level  for  which  the 

FACTOR:  standard  applies. ] 

SUBFACTOR: 

DESCRIPTION:  [Describe  the  particular  Item, 

Factor  or  Subfactor  to  be  evaluated. ] 
STANDARD:  This  standard  is  met  when  - 

[Indicate  those  requirements  from  the 

RFP  SOW,  and  specifications  that  the 

evaluator  will  look  for  in  the  offeror's 
proposal  to  determine  if  the  standard  is 


Examples  of  R&M  standards  contained  in  this  guide 
provide  effective  wording  and  critical  R&M  requirements 
that  must  be  shown  in  an  offeror's  proposal  to  demonstrate 
the  particular  R&M  capability.  The  examples  should  be 
tailored  to  the  system  being  acquired  by  the  user  of  this 
guide.  The  examples  are  also  useful  in  providing  foresight 
to  those  involved  in  developing  an  R&M  SOW,  and  evaluation 
criteria  as  any  critical  R&M  requirement  in  a  standard  must 
also  be  included  in  the  RFP.  As  stated  earlier,  standards 
are  tied  directly  to  the  evaluation  criteria,  SOW,  and 
specifications.  Therefore,  if  critical  R&M  requirements 
are  not  included  in  these  areas,  they  may  not  be  used  as 
source  selection  standards. 

The  examples  in  the  guide  do  not  reflect  R&M  as  an 
item  under  the  technical  area  as  current  guidance  suggests. 
Programs  utilizing  the  current  emphasis  of  R&M  as  the 
number  one  item  under  the  technical  area  are  so  new  that 
standards  were  unavailable  as  they  are  treated  as  source 
selection  sensitive.  This  in  no  detracts  from  the  useful¬ 
ness  of  the  examples  as  they  include  R&M  requirements  which 
must  be  demonstrated  by  the  offeror  whether  R&M  is  treated 
as  an  item,  factor  or  subfactor. 

The  following  examples  illustrate  elaborate  R&M 
standards  which  may  be  tailored  to  a  major  weapon  system 
acquisition. 


Elaborate  standards 


Example  1: 

AREA:  Technical 

ITEM:  Functional  Programs 

FACTOR:  Reliability — This  factor  addresses  the 
engineering  tasks  required  for  the  reliability  program  at 
the  item  level. 

STANDARDS:  The  offeror’s  technical  proposal,  with 
respect  to  reliability,  will  be  technically  acceptable  when 
the  following  standards  are  met: 

1.  The  offeror  shows  a  clear  understanding  of  the 
reliability  program  requirements,  including  program  plan, 
reliability  predictions,  reliability  allocation,  demonstra¬ 
tion  of  MTBM  through  operational  verification,  and  a 
meaningful  failure  control  program.  The  offeror  identifies 
in  a  clear  manner  any  problems  associated  with  the  relia¬ 
bility  program. 

2.  The  offeror  outlines  his  approach  to  a  logical 
and  practical  reliability  program  which  covers  the  required 
aspects  of  reliability,  such  as  an  approach  to  the  overall 
reliability  program,  scope  of  program  to  include  relia¬ 
bility  predictions,  allocations,  demonstration,  and  failure 
control.  Identifies  any  STDs/HDBKs/etc. ,  to  be  used  as  an 
aid  in  meeting  SOW/TRD  requirements  and  shows,  in  general, 
how  the  STDs/HDBKs/etc.,  will  be  used  in  the  area  of  relia¬ 
bility  predictions  and  allocations. 


3.  The  offeror  shows  in  clear  and  concise  manner 
that  the  proposed  reliability  program  complies  with  the 
requirements  of  the  SOW/TRD/DIDs . 

Example  2: 

AREA:  Technical 

ITEM:  Functional  Programs 

FACTOR:  Maintainability — This  factor  addresses  the 
engineering  tasks  required  for  the  maintainability  program 
at  the  item  level. 

STANDARDS:  The  offeror's  technical  proposal,  with 
respect  to  maintainability,  will  be  technically  acceptable 
when  the  following  standards  are  met: 

1.  The  offeror  shows  a  clear  understanding  of  the 
maintainability  program  requirements,  including  a  program 
plan  to  insure  that  MTTR  requirements  will  be  met,  a  satis¬ 
factory  method  for  predicting  item  MTTR,  allocation  of  MTTR 
requirements,  and  a  demonstration  technique  for  proving 
compliance  with  MTTR  requirements.  Also,  the  offeror 
outlines  in  a  clear  manner  a  method  of  resolving  any 
problems  which  could  arise  as  work  progresses  in  the  main¬ 
tainability  area. 

2.  The  offeror  outlines  his  approach  to  a  logical 
and  practical  maintainability  program  which  covers  the 
required  aspects  of  maintainability,  including  an  approach 
to  the  overall  maintainability  program,  what  handbooks  and 


other  aids  will  be  used  for  MTTR  prediction,  how  these  aids 
will  be  used,  how  partitioning  may  be  used  to  allocate  MTTR 
to  stay  within  TRD  values,  and  also  their  approach  to  demon 
strating  the  MTTR  requirements. 

3.  The  offeror  shows  in  a  clear  and  concise  manner 
that  the  proposed  maintainability  program  complies  with  the 
requirements  of  the  SOW/TRD/DIDs . 

Example  3: 

Reliability.  Description:  This  factor  addresses 
the  engineering  tasks  required  for  the  reliability  program 
at  the  item  level. 

Approach:  The  offeror  outlines  his  approach  to  a 
logical  and  practical  reliability  program  which  covers  the 
required  aspects  of  reliability,  such  as  an  approach  to  the 
overall  reliability  program,  scope  of  program  to  include 
reliability  predictions,  allocations,  demonstration,  and 
failure  control.  Identifies  any  STDs/HDBKs/etc. ,  to  be 
used  as  an  aid  in  meeting  SOW/TRD  requirements  and  shows, 
in  general,  how  the  STDs/HDBKs/etc.,  will  be  used  in  the 
area  of  reliability  predictions  and  allocations. 

This  standard  is  met  when: 

1.  The  offeror  identifies  a  satisfactory  organiza¬ 
tion  to  implement  and  complete  the  reliability  program 
tasks  (including  experienced  personnel). 


2.  The  offeror  outlines  a  sound  reliability  program 
per  applicable  MIL  STDs. 

3.  The  offeror  identifies  satisfactory  STDs/HDBKs/ 
etc.,  to  assist  the  following: 

a.  Prediction  techniques 

b.  Allocation  to  SRU 

c.  Demonstration  requirement 

d.  Failure  assessment. 

4.  The  offeror  outlines  a  sound  approach  to 
demonstrating  MTBM  from  "first  test"  through  "IOT&E. " 

5.  The  offeror  outlines  a  sound  approach  to 
detecting,  reporting  and  analyzing/correcting  pattern 
failures . 

Compliance:  The  offeror  shows  in  a  clear  and  concise 

manner  that  the  proposed  reliability  program  complies  with 
the  requirements  of  the  SOW/TRD/DIDs . 

This  standard  is  met  when: 

1.  The  offeror's  proposal  complies  with  the  major 
aspects  of  the  assignment,  evaluation,  and  demonstration  of 
reliability  requirements. 

2.  The  offeror  provides  a  preliminary  plan  for  the 
proposed  reliability  program  which  complies  with  the 
respective  SOW/TRD  paragraph  requirements.  These  include: 

a.  Reliability  predictions 

b.  Reliability  allocation 

c.  Reliability  demonstration 


d.  Failure  reporting  mechanism 

Understanding:  The  offeror  shows  a  clear  under¬ 
standing  of  the  reliability  program  requirements,  including 
program  plan,  reliability  predictions,  reliability  alloca¬ 
tion,  demonstration  of  MSBM  through  operational  verifica¬ 
tion  and  a  meaningful  failure  control  program.  The  offeror 
identifies  in  a  clear  manner  any  problems  associated  with 
the  reliability  program. 

This  standard  is  met  when: 

1.  The  offeror  shows  an  adequate  plan  for  insuring 
that  reliability  program  plan  requirements  are  met.  This 
includes  discussing  the  relationship  between  elements,  such 
as: 

a.  Reliability  predictions 

b.  Reliability  allocations 

c.  Reliability  demonstrations 

d.  Failure  of  equipment/correction. 

2.  The  offeror  shows  a  good  understanding  of  how 
various  data  sources  and  experience  with  reliability  predic¬ 
tion  techniques  are  used  to  develop  reasonable  predictions 
that  can  be  met  by  the  type  of  equipment  selected/ 
identified. 

3.  The  offeror  understands  the  level  of  effort  neces¬ 
sary  to  assess  the  impact  of  various  tradeoffs  on  system/ 
LRU  reliability. 


4.  The  offeror  addresses  the  adequacy  of  schedules/ 
plans/programs  dealing  with  reliability,  including  the 
major  factors  that  typically  limit  a  viable  reliability 
program. 

Example  4: 

Maintainability.  Description:  This  factor 
addresses  the  engineering  tasks  required  for  the  maintain¬ 
ability  program  at  the  item  level. 

Approach:  The  offeror  outlines  his  approach  to  a 
logical  and  practical  maintainability  program  which  covers 
the  required  aspects  of  maintainability,  including  an 
approach  to  the  overall  maintainability  program,  what  hand¬ 
books  and  other  aids  will  be  used  for  MTTR  prediction,  how 
these  aids  will  be  used,  how  partitioning  may  be  used  to 
allocate  MTTR  to  stay  within  TRD  values,  and  also  their 
approach  to  demonstrating  the  MTTR  requirements. 

This  standard  is  met  when: 

1.  The  offeror  identifies  an  organization  which 
should  adequately  handle  the  maintainability  program, 
including  experienced  personnel. 

2.  The  offeror  outlines  a  sound  maintainability 
program  per  applicable  MIL  STDs.  This  includes  corrective/ 
preventive  maintenance,  servicing,  tools,  test  equipment, 
and  skilled  personnel. 


3.  The  offeror  has  proposed  to  use  satisfactory 
STDs/HDBKs/etc. ,  to  scope  maintainability  in  the  following 
areas: 

a.  MTTR  predictions 

b.  MTTR  allocation  to  SRU 

c.  Demonstration  requirement. 

4.  The  offeror  shows  a  sound  approach  to  demon¬ 
strating  maintainability  requirements,  through  verification 
of  maintainability  parameters  (e.g.,  Mcfc,  Mpt,  etc.). 

5.  The  offeror  shows  adequate  schedules/plans/ 
programs  dealing  with  the  maintainability  area,  including 
any  major  factors  that  typically  limit  a  viable  maintain¬ 
ability  program. 

Compliance:  The  offeror  shows  in  a  clear  and  concise 
manner  that  the  proposed  maintainability  program  complies 
with  the  requirements  of  the  SOW/TRD/DIDs . 

This  standard  is  met  when: 

1.  The  offeror’s  proposal  complies  with  all  major 
elements  of  maintainability  from  prediction  through  demon¬ 
stration. 

2.  The  offeror  provides  a  preliminary  plan  for  the 
proposed  maintainability  program  which  complies  with  the 
respective  SOW/TRD  paragraph  requirements. 

a.  Maintainability  predictions  and  allocation 

b.  Maintainability  analysis 


c.  Maintainability  demonstration 

d.  MTTR  and  preventive  maintenance  values 

e.  Interface  between  maintainability  require¬ 
ments  and  maintenance  levels/tools/test  equipment/ 
personnel/etc. 

Understanding:  The  offeror  shows  a  clear  under¬ 
standing  of  the  maintainability  program  requirements, 
including  a  program  plan  to  insure  that  MTTR  requirements 
will  be  met,  a  satisfactory  method  for  predicting  item 
MTTR,  allocation  of  MTTR  requirements,  and  a  demonstration 
technique  for  proving  compliance  with  MTTR  requirements. 
Also,  the  offeror  outlines  in  a  clear  manner  a  method  of 
resolving  any  problems  which  could  arise  as  work  progresses 
in  the  maintainability  area. 

This  standard  is  met  when: 

1.  The  offeror  shows  an  adequate  plan  for  insuring 
that  maintainability  program  plan  requirements  are  met. 

This  includes  discussing  the  relationship  between  elements 
such  as: 

a.  MTTR  predictions 

b.  MTTR  allocation 

c.  Maintainability  demonstration 

d.  Maintenance  levels 

e.  Corrective  and  preventive  maintenance. 
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2.  The  offeror  understands  the  need  for,  and  has 
identified  satisfactory  data  sources  to  assist  in  making 
reasonable  maintainability  predictions. 

3.  The  offeror  proves  his  understanding  of  maintain¬ 
ability  tradeoffs  through  item  partitioning,  as  related  to 
the  maintenance  levels. 

4.  The  offeror  understands  the  procedures  involved 
in  conducting  a  maintainability  demonstration  and  of  the 
responsibilities  of  government  and  contractor  in  performing 
the  demonstration. 

Specific  Standards 

The  following  R&M  standards  address  specific  aspects 
of  R&M  and  utilize  various  references  to  SOW  paragraphs 
which  the  offeror  must  comply  with  in  order  to  meet  the 
standards.  The  user  of  this  guide  must  tailor  specific  SOW 
paragraphs. 

Example  1: 

AREA:  Technical 

ITEM:  T.l  System  Design/Performance 

FACTOR:  T. 1.19  Maintainability/Testability 

DESCRIPTION:  This  factor  involves  the  review  of  the 
BITE,  Testability,  Test  points,  and  accessibility  of  the 
design. 


103 


STANDARDS:  The  standards  will  be  met  when  the 
offeror  indicates  compliance  with  the  following  areas: 

1.  Maintainability  design  Specification  paragraph 
3. 2.4.2 

2.  Test  points  Specification  paragraph  3. 2. 4. 2.1 

3.  Accessibility  Specification  paragraph  3. 2. 4. 2. 2 

4.  BIT  Specification  paragraph  3. 2. 4. 2. 3  and 
3. 2. 4. 2. 3.1 

5.  Skill  level  paragraph  3. 2. 4. 3. 

Example  2: 

AREA:  Technical 

ITEM:  T.l  System  Design/Performance 

FACTOR:  T.l. 18  Reliability  Prediction 

DESCRIPTION:  This  factor  involves  the  review  of  the 
offeror's  reliability  prediction. 

STANDARDS:  The  standards  will  be  met  when  the 
proposal  defines  the  following  areas ,  but  not  limited 
specifically  to: 

1.  The  Reliability  Prediction  exceeds  the  required 
Specification  paragraph  3. 2. 3.1 

2.  The  worst  case  environmental  conditions  are  used 
to  calculate  the  Prediction  Reliability  values  per  SOW 
paragraph  1050.4.1. b 

3.  A  realistic  parts  count,  quality,  and  stress 
level  is  used  per  SOW  paragraph  1050.4.1.b. 


AREA:  Technical 

ITEM:  T. 1  System  Design/Performance 

FACTOR:  T.1.17  Reliability 

DESCRIPTION:  This  factor  involves  the  review  of  the 
offeror's  reliability  work  effort. 

STANDARDS:  The  standards  are  met  when  sufficient 
information  is  provided  to  establish  high  confidence  in  the 
compatibility  of  the  offeror's  reliability  work  effort  to 
the  system  specification  requirements.  This  information 
should  include,  but  is  not  specifically  limited  to: 

1.  The  approach  by  which  the  offeror  plans  on 
conducting  the  requirements  of  SOW  paragraphs  1050. 4. l.c 
and  d 

2.  The  general  compliance  of  the  offeror  with  ITO 
paragraph  2.2. 

Example  4: 

AREA:  Technical 

ITEM:  T.l  System  Design/Performance 

FACTOR:  T.l. 16  Reliability  and  Maintainability 
Program  Plans 

DESCRIPTION:  This  factor  involves  the  review  of  the 
offeror's  reliability  and  maintainability  program  plans. 


STANDARDS:  The  standards  are  met  when  the  proposal 
defines  the  following  areas ,  but  is  not  limited 
specifically  to: 

1.  Compliance  with  SOW  paragraph  1050.4  and 
1050.4.1a,  and  ITO  2.2 

2.  Compliance  with  Specification  paragraph  3.2.3  and 

3.2.4 

3.  Reporting  structure  of  R&M  engines 

4.  Corporate  policies  that  impart  R&M  so  as  to 
indicate  what  priority  it  has. 

5.  Size  of  R&M  departments,  how  many  engineers  will 
be  assigned  to  the  program  and  their  experience  levels. 

6.  Reasonableness  of  schedules,  milestones, 
reporting,  review  of  problem  areas,  and  design  guidelines. 

Example  5: 

AREA:  Technical 

ITEM:  T. 1  System  Design/Performance 

FACTOR:  T.1.20  Maintainability  Prediction 

DESCRIPTION:  This  factor  involves  the  review  of  the 
Maintainability  Prediction. 

STANDARDS:  The  standards  will  be  met  when  the 
offeror  indicates  compliance  with  the  following  areas: 

1.  The  Maintainability  requirements  of  the 
Specification  paragraph  3. 2.4.1 


2.  Maintainability  requirements  of  SOW  paragraph 


1050. 4. l.b  and  c. 

Boiler  Plate  Standards 

The  following  examples  are  in  a  "boiler  plate"  format 
and  may  be  tailored  to  a  specific  system  by  referencing 
appropriate  paragraphs  from  the  RFP. 

Example  1: 

AREA:  Technical 
ITEM:  Reliability 

FACTOR:  Program  Plan 

DESCRIPTION:  This  factor  is  to  evaluate  the 

offeror's  reliability  program  elements  of  MIL  STD  785 , 
which  describe  the  approach  and  method  of  accomplishing  the 
reliability  requirements  of  the  RFP. 

STANDARD:  This  standard  is  satisfied  when: 

1.  The  offeror  has  indicated  complete  understanding 
and  intent  to  comply  with  all  reliability  requirements  of 
the  RFP. 

2.  The  offeror  defines  methods  and  procedures  for 
meeting  the  reliability  program  tasks  required  by  the  RFP. 
This  includes  methods  by  which  the  offeror  will  ensure  that 
the  quantitative  reliability  requirements  are  achieved  or 
sufficient  justification  is  given  for  tradeoffs  proposed 
and  deviations  requested. 


3.  Any  proposed  expansions  of  the  Development 
Specification  requirements  are  satisfactory  and  do  not 
reduce  the  basic  specification  requirements. 

4.  The  offeror  proposes  a  sound  and  effective 
reliability  program  which  defines  the  approach  and 
describes  the  method  for  predicting  and  allocating  system 
reliability  parameters,  and  optimizing  reliability  to 
obtain  maximum  availability  at  the  lowest  life  cycle  cost. 

RFP  REFERENCE: 

Example  2: 

AREA:  Technical 

ITEM:  Maintainability 

FACTOR:  Design  Approach 

DESCRIPTION:  This  factor  is  to  assess  the  design 
approach  to  be  applied  by  the  offeror  in  meeting  the 
quantitative  maintainability  requirements  as  well  as 
accomplishing  the  qualitative  maintainability  requirements 
to  insure  consonance  with  operational  support  concept. 

STANDARD:  This  standard  is  satisfied  when: 

1.  The  offeror  has  indicated  complete  understanding 
and  intent  to  comply  with  all  maintainability  requirements 
of  the  RFP. 

2.  The  offeror  proposes  a  sound  and  effective 
maintainability  program  which  defines  the  approach  and 
describes  the  method  for  predicting  and  allocating 


maintainability  parameters  to  obtain  maximum  availability 
at  lowest  life  cycle  cost. 


3.  The  offeror's  proposed  inputs  to  the  maintenance 
concept  ensure  achievement  of  availability  goals  as 
described  in  the  operational  concept  in  the  ICS  SOW. 

4.  The  offeror  specifies  and  substantiates  effective 
support  elements  that  will  satisfy  the  operational  concept. 
This  includes  the  following: 

a.  Levels  of  maintenance 

b.  Manhour /ski 11  requirements 

c.  Downtime 

d.  Servicing  requirements 

e.  Checkout 

f.  Calibration/alignment  procedures 

g.  Inspections. 

RFP  REFERENCES: 

Evaluator  Questions 

The  following  questions  may  be  used  by  evaluators  to 
further  clarify  if  an  offeror  has  demonstrated  the  capa¬ 
bility  to  perform  the  R&M  requirements  in  the  RFP.  The 
questions  cover  four  areas:  (1)  Reliability  Program  Plan, 
(2)  Maintainability  Program  Plan,  (3)  Built-In-Test  (BIT), 
and  (4)  Reliability  Testing. 


Reliability  Program  Plan 


1.  Has  the  offeror  explained  how  the  reliability 
and  maintainability  (R&M)  requirements  and  concepts  will  be 
incorporated  into  the  design  and  development  process  (i.e., 
design  reviews,  specification  reviews,  checklists,  guide¬ 
lines,  etc.)?  Have  the  R&M  engineers'  role  and  authorita¬ 
tive  responsibilities  been  defined? 

2.  Does  the  offeror  intend  to  train  other  depart¬ 
mental  managers  of  the  company  in  the  reliability 
principles,  requirements,  and  goals? 

3.  Has  the  offeror  identified  if  it  is  the  role  of 
the  reliability  and  maintainability  engineers  or  mangers  to 
have  final  drawing  and  specification  sign-off  responsibili¬ 
ties  before  formal  design  release? 

4.  Does  the  offeror  provide  details  concerning  the 
failure  review  board  (who  is  a  member,  how  often  does  it 
meet,  what  are  the  topics)?  Has  the  offeror  included  a 
System  Safety  engineer  and  a  representative  of  manufac¬ 
turing  as  a  participating  member  of  the  failure  review 
board? 

5.  How  does  the  offeror's  stress  analysis  and 
predictions  computer  program  relate  and  compare  to  the 
prediction  techniques  data  within  MIL  HDBK  217d?  Will  the 
offeror's  design  requirements  group  comply  with  (as  a 
minimum)  the  derating  requirements  of  AS-4613  and  MIL  HDBK 


217D? 


6.  Has  the  offeror  complied  with  the  reliability/ 


maintainability  definitions  and  failure  definitions  in 
accordance  with  MIL  STD  721C  and  MIL  STD  781C? 

7.  Will  the  offeror's  reliability  test  data  and  log 
books,  etc.,  be  made  available  to  the  Government  Representa 
tive  (Procuring  Activity)  at  his/her  request? 

8.  Has  the  offeror  submitted  the  reliability  growth 
rate  for  each  LRU  type  and  the  system  and  submitted  the 
growth  rates  with  the  associated  growth  curves  for  the 
system  and  each  LRU? 

9.  Has  the  offeror  interfaced  and  used  the  failure 
modes,  effects,  and  critical  analysis  (FMECA)  in  the  system 
safety  program  and  analysis?  Has  the  offeror  indicated 
that  FMECA  will  be  performed  in  accordance  with  MIL  STD 
1629A? 

10.  Has  the  offeror's  parts  control  program 
addressed  and  included  that  the  program  will  be  performed 
in  accordance  with  MIL  STD  965? 

11.  Has  the  offeror  provided  plans  and  procedures  to 
translate  and  analyze  the  contractual  Mean-Time-Between- 
Failure  (MTBF3 )  to  operational  Mean-Time-Between- 
Maintenance  (MTBMs);  type  1 — Inherent,  type  2 — Induced, 
type  6 — No  defect,  total;  values?  What  are  the  MTBM,  MTTR, 
MCTMAX,  and  Mean-Downtime  definitions? 

12.  Has  the  offeror  determined  major  decision 
factors  for  use  in  evaluating  a  trade-off  study  for  a  new 
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design  or  redesign?  Has  the  offeror  identified  procedures 
and  evaluation  techniques  in  this  type  of  decision  making? 

13.  Has  the  offeror  provided  source  data  to  support 
and/or  back  up  the  field  reliability  MTBF  figures  stated 
within  the  proposal  as  proof  of  past  experience?  Where  did 
the  data  come  from?  How  were  the  MTBFs  calculated?  How 
was  the  data  collected  and  tracked? 

14.  Within  the  offeror's  proposed  reliability 
program  plan,  have  reliability  modeling,  reliability  predic 
tions,  reliability  allocations  been  addressed  as  required 
by  MIL  STD  785B? 

Reliability  Testing 

1.  Has  the  offeror  provided  information  on  what  type 
of  support  equipment  will  be  utilized  to  support  ESS,  RDT, 
RQT,  and  flight  test? 

2.  Does  the  offeror  plan  to  conduct  burn-in  testing 
(optional  for  contractor  to  perform)  on  all  the  system  LRUs 
prior  to  environmental  stress  screening  (ESS)? 

3.  If  the  offeror  plans  to  perform  burn-in,  is  it 
indicated  how  the  failures  found  during  the  burn-in  testing 
will  be  documented  and  tracked? 

4.  Has  the  offeror  indicated  whether  reliability 


qualification  testing  will  be  conducted  on  a  system  or  LRU 


5.  Has  the  offeror  indicated  if  the  same  test  LRUs 
(systems)  used  in  RDT  will  be  used  in  RQT  and  maintain¬ 
ability  demonstration? 

6.  Has  the  offeror  provided  the  types  and  quantities 
of  test  set  chambers  to  run  the  ESS,  ROT,  and  RQT  testing? 

7.  Has  the  offeror  explained  the  ESS,  RDT,  and  RQT 
testing?  Have  testing  plans,  procedures,  test  duration, 
failure  definitions,  test  profile,  documentation,  tracking, 
and  corrective  actions  of  failures  been  provided? 

8.  Has  the  offeror  provided  a  detailed  calculation 
showing  how  RQT  and  RDT  test  time  was  derived? 

9.  Has  the  offeror  explained  the  plans  to  mount  the 
test  samples  (system)  into  the  test  set  chamber? 

Built-In-Test  (BIT) 

1.  Is  the  offeror's  BIT  program  in  compliance  with 
MIL  STD  415? 

2.  Has  offeror  indicated  the  BIT  will  be  tested 
under  environmental  qualification  testing,  ESS,  RDT,  and 
RQT? 

3.  Will  the  offeror's  BIT  be  operational  during  ESS, 
RDT,  and  RQT  testing? 

4.  Has  the  offeror  indicated  what  will  be  done  if 
the  BIT  of  the  system  indicates  that  there  is  a  failure 
within  the  system  and  the  automatic  test  set(s)  indicates 
there  is  no  failure?  Or  vice  versa? 


5.  Has  the  offeror  indicated  how  the  proposed  BIT 
values  were  calculated  and  developed? 

6.  Does  the  offeror  indicate  how  BIT  failure  to 
detect  and  isolate  are  documented  and  tracked  and  to  whom 
the  documentation  is  submitted? 

7.  Does  the  offeror  indicate  how  the  BIT  fault  data 
will  be  displayed  at  the  LRU,  SRU,  and  system  levels? 

8.  Does  offeror’s  BIT  monitor  and  track  all  the 
redundant  bases  in  the  system? 

9.  Will  the  offeror's  BIT  monitor  all  the  software 
within  the  system  to  detect  and  isolate  software  problems? 

Maintainability  Program  Plan 

1.  Has  the  offeror  developed  preliminary  MTTR  and 
MCTMAX  predictions?  Are  figures  at  the  SRU  and  LRU  levels 
and  the  procedures  used  in  obtaining  them  provided? 

2.  Does  the  offeror  have  maintainability  and  their 
associated  studies  and  tests  a  major  decision  factor  in  the 
trade-off  studies  of  each  design  consideration  or  redesign 
change? 

3.  Does  the  offeror  indicate  who  will  have  final 
authority  on  conducting  the  maintainability  demonstration? 

4.  Has  the  offeror  provided  the  scheduled  and 
unscheduled  maintenance  concept? 


Difficulty  m  Evaluating  r&m 


The  SOW  tasks  such  as  test  plans  and  specifications 
are  easier  R&M  requirements  to  evaluate.  Evaluation  of 
data  submitted  is  a  difficult  area  to  evaluate.  At  times, 
contractors  submit  volumes  of  data.  It  is  up  to  the 
evaluator  to  determine  if  the  data  is  reasonable  or  makes 


sense. 


Proposals  submitted  during  the  conceptual  phase  of 
acquisition  are  also  difficult  to  evaluate  as  requirements 
are  vague.  There  is  often  no  past  information  available 
from  which  to  judge  the  reasonableness  of  proposals. 

Future  capability  is  difficult  to  evaluate.  When  a  system 
will  not  be  produced  for  5-10  years,  it  must  be  determined 
if  the  contractor  will  have  sufficient  capability  in  the 
future.  Many  uncertainties  exist  in  evaluating  this  area. 

Evaluation  of  R&M  during  the  full  scale  development 
and  production  phases  of  acquisition  are  easier.  During 
these  phases,  R&M  tasks  and  requirements  are  fully  identi¬ 
fied;  therefore,  evaluators  know  what  contractors  need  to 
provide  in  proposals  to  adequately  show  R&M  capability. 

Contractor  analysis  of  R&M  includes  identification  of 
critical  R&M  items  in  the  system.  It  is  difficult  to 
dispute  the  selection  of  critical  items  by  the  contractor. 
Also,  in  order  to  evaluate  critical  items,  the  evaluator 
must  be  familiar  with  the  data  submitted  along  with 


procedures  used  to  collect  the  data  and  field  maintenance 
procedures.  Contractor  R&M  predictions  also  involve  an 
indepth  evaluation  and  are  often  difficult  to  dispute. 

Maintainability  in  general  is  a  difficult  area  to 
evaluate  as  it  is  difficult  to  conceptualize  maintain¬ 
ability  on  paper.  It  often  takes  hands-on  experience  with 
a  system  to  identify  maintainability  problems.  Evaluators 
must  often  rely  on  data  and  designs  submitted  to  determine 
the  maintainability  of  the  system. 

Past  performance  of  a  contractor  on  R&M  requirements 
is  difficult  to  evaluate  as  information  is  often  not  avail¬ 
able  in  the  source  selection  on  R&M  past  performance. 

The  evaluation  of  R&M  budgeted  growth  curves  is  diffi 
cult.  Often  times,  determination  of  start  points  and 
slopes  are  a  matter  of  judgement.  Data  is  not  available  to 
refute  the  contractor's  submission. 

Conclusion 

Through  the  use  of  this  guide,  personnel  will  be 
aided  in  the  effective  development  of  R&M  evaluation 
criteria  and  standards.  Today,  more  than  ever,  with  the 
increasing  cost  and  complexity  of  weapon  systems,  the  U.S. 
Air  Force  must  insure  that  the  most  reliable  and  maintain¬ 
able  systems  are  acquired  for  the  dollars  spent. 


Bibliography 


Aeronautical  Systems  Division  (AFSC) .  The  Source 
Selection  Process.  A  handbook  for  use  by  ASD 
personnel.  Wright-Patterson  APB  OH:  HQ  ASD,  AFSC, 
June  1981. 

Armstrong,  Robert,  R&M  Engineer.  Personal  interview 
HQ  ASD/AFES,  Wright-Patterson  AFB  OH,  26  July  1985. 

Detert,  Wallace,  R&M  Engineer.  Personal  interview. 
HQ  ASD/TAES,  Wright-Patterson  AFB  OH,  22  July  1985. 

Liberti,  1st  Lt  William,  USAF,  R&M  Engineer. 

Personal  interview.  HQ  ASD/RWEX,  Wright-Patterson 
AFB  OH,  26  July  1985. 

Lucka,  William,  Division  Chief.  Personal  interview. 
HQ  ASD/ENSI,  Wright-Patterson  AFB  OH,  17  July  1985. 

McLennan,  Roxley  K.  The  Feasibility  of  a  Decision 
Support  System  for  the  Determination  of  Source 
Selection  Evaluation  Criteria.  MS  thesis,  LSSR 
84S-42.  School  of  Systems  and  Logistics,  Air  Force 
Institute  of  Technology  (AD) ,  Wright-Patterson  AFB 
OH,  September  1984  (AD-A147  655). 

Neiman,  Clay,  R&M  Engineer.  Personal  interview.  HQ 
ASD/AEES,  Wright-Patterson  AFB  OH,  3  July  1985. 

Papenbrock,  Richard,  Lead  Engineer,  Product 
Assurance.  Personal  interview.  HQ  ASD/RWEX, 
Wright-Patterson  AFB  OH,  26  July  1985. 

Read,  Robert,  R&M  Engineer.  Personal  interview. 
AFALC,  Wright-Patterson  AFB  OH,  2  August  1985. 

Secen,  1st  Lt  Michael,  USAF,  R&M  Engineer.  Personal 
interview.  AFALC,  Wright-Patterson  AFB  OH,  17  July 
1985. 

Skantze,  Gen  Lawrence  A.,  USAF.  "Emphasizing 
Reliability,  Maintainability,  and  Producibility  in 
the  Design  Process.”  Headquarters  Air  Force  Systems 
Command  letter,  21  December  1984. 

U.S.  Department  of  Defense.  Maintainability  Program 
for  Systems  and  Equipment.  MIL  STD  470A. 

Washington  DC:  Government  Printing  Office,  3  January 


13. 


.  Major  Systems  Acquisition  Procedures.  DOD 
Instruction  5000.2.  Washington  DC:  Government 
Printing  Office,  3  May  1983. 

14.  - .  Major  Systems  Acquisitions.  DOD  Directive 

5000.1.  Washington  DC:  Government  Printing  Office,  29 
March  1982. 


15.  - .  Reliability  and  Maintainability.  DOD 

Directive  5000.40.  Washington  DC:  Government  Printing 
Office,  8  July  1980. 


16.  - .  Reliability  Program  for  Systems  and  Equipment 

Development  and  Production.  MIL  STD  785B. 

Washington  DC:  Government  Printing  OFfice,  15 
September  1980. 

17.  - .  Selection  of  Contractual  Sources  for  Major 

Defense  Systems .  DOD  Directive  4105.62.  Washington 
DC:  Government  Printing  Office,  6  January  1976  (and 
amended  3  March  1977). 


18.  U.S.  Department  of  the  Air  Force.  Air  Force 
Reliability  and  Maintainability.  AFR  800-18. 

Wash ington  DC :  Government  Printing  Office,  15  June 
1982. 


19.  - — .  Briefing  Formats  for  Source  Selection  Actions 

(AFR  70-15,  22  February  1984).  Wright-Patterson  AFB 
OH:  Headquarters  Aeronautical  Systems  Division 
(AFSC),  5  June  1985. 

20.  - .  Reliability  and  Maintainability  Action  plan 

R&M  2000~  Washington  DC:  Government  Printing  Office, 
1  February  1985. 

21.  - .  Reliability  and  Maintainability  of  Air  Force 

Weapon  Systems  -  Action  Memorandum.  Washington  DC: , 
17  September  1984. 

22.  - .  Source  Selection  Plan  Preparation  Guide. 

ASDP  800.7.  Wright-Patterson  AFB  OH:  Headquarters 
Aeronautical  Systems  Division  (AFSC),  June  1985. 

23.  - .  Source  Selection  Policy  and  Procedures.  AFR 

70-15.  Washington  DC:  Government  Printing  Office,  22 
February  1984. 


118 


.  '•  j  J.  >  -  l’: 


Source  Selection  Policy  and  Procedures 


Interim  ASD  Supplement  1  to  APR  70-15.  Wright 
Patterson  AFB  OH;  Headquarters  Aeronautical  Systems 
Division  (AFSC),  21  March  1985. 

Wolanski,  Gene,  R&M  Engineer.  Personal  interview. 
ASD/ENSI,  Wright-Patterson  APB  OH,  10  July  1985. 


Captain  Danial  E.  Harnen,  Jr.  was  born  on  31  July 
1959  in  Patterson,  New  Jersey.  He  was  educated  at  schools 
in  New  Jersey  and  graduated  from  Bulter  High  School  in 
1977.  He  graduated  from  Norwich  University  in  1981  with 
the  degree  of  Bachelor  of  Science  in  Business 
Administration.  Upon  graduation  from  Norwich  University, 
he  was  commissioned  a  Second  Lieutenant  in  the  United 
States  Air  Force. 

Prior  to  entering  the  School  of  Systems  and 
Logistics,  U.S.  Air  Force  Institute  of  Technology  in  May 
1984,  he  was  assigned  at  Channute  Air  Force  Base,  Illinois 
where  he  served  as  a  buyer  and  contract  administrator  at 
the  base  Contracting  Office. 

Permanent  Address:  1386  Sanzon  Drive 

Fairborn  OH  45324 


UNCLASSIFIED 


1«.  REPORT  SECURITY  CLASSIFICATION 

UNCLAS  S I F I ED _ 


2a  security  classification  authority 


o*so 


REPORT  DOCUMENTATION  PAGE 


lb.  RESTRICTIVE  MARKINGS 


2b.  OECLASSIFICATION/OOWNGRAOING  SCHEDULE 


4.  PERFORMING  ORGANIZATION  REPORT  NUMBER(S) 

AFIT/GLM/LSQ/8 5  S-30 


3.  OISTRI8UTION/AVAI  LABILITY  OF  REPORT 

Approved  for  public  release; 
distribution  unlimited. 


S.  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 


6a.  NAME  OF  PERFORMING  ORGANIZATION  16b.  OFFICE  SYMBOL  7a  NAME  OF  MONITORING  ORGANIZATION 

School  of  Systems  and 

Logistics  AFIT/LSQ 


6c.  AOORESS  (City,  Slat*  and  ZIP  Coda) 

Air  Force  Institute  of  Technology 
Wright-Patterson  AFB,  Ohio  45433 


7b.  AOORESS  (City,  State  and  ZIP  Coda) 


Sa  NAME  OF  FUNOING/SPONSORING 
ORGANIZATION 


8b.  OFFICE  SYMBOL  9.  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 
(If  applicable) 


8c.  AOORESS  (City.  State  and  ZIP  Code) 

10.  SOURCE  OF  FUNDING  NOS. 

PROGRAM 
ELEMENT  NO. 

PROJECT 

NO. 

TASK 

NO. 

WORK  UNIT 
NO. 

11.  TITLE  (Include  Security  Claeeiflcation) 

See  Box  19 

12.  PERSONAL  AUTHORIS) 

Daniel  E.  Harnen,  Jr..  Captain.  USAF 


13a  TYPE  OF  REPORT 

MS  Thesis 


IB.  SUPPLEMENTARY  NOTATION 


13b.  TIME  COVEREO 

14.  DATE  OF  REPORT  (Yr..  Mo.,  Day)  | 

FROM  TO 

IS.  PAGE  COUNT 


17.  COSATI  COOES 


FIELD  I  GROUP  I  SUB.  GR. 


14  I  04 


18.  SUBJECT  TERMS  (Con tin ua  on  raoana  if  neceeeary  and  identify  by  block  number) 

Source  Selection.  Evaluation  Criteria,  Evaluation 
Standards,  Reliability,  Maintainability 


IB.  ABSTRACT  ( Continue  on  raoarea  if  nacaaaary  and  identify  by  block  number) 

Title:  DEVELOPING  SOURCE  SELECTION  EVALUATION  CRITERIA  AND  STANDARDS 

FOR  RELIABILITY  AND  MAINTAINABILITY 

Thesis  Chairman:  Roy  R.  Wood,  Jr. 

Professor  of  Quantitative  Methods 


flavored  tor,  pylic  ralocaa:  1AW  APR  1M-W, 

WOLAVXH  ^ Soft 

Dean  lot  Roaocieb  and  Protoaoionat  Daaaiopaoat 
Ail  Faraa  lutiiuU  oi  Taahnoloaj  (Mb. 
IVttoma  ATI  OB  IMS 


20.  OISTRIBUTION/AVAILABILITY  of  abstract 


21.  ABSTRACT  SECURITY  CLASSIFICATION 


UNCLASSIPIEO/UNLIMITEO  25  SAME  AS  RPT.  □  OTIC  USERS  □  UNCLASSIFIED 


22a.  NAME  OP  RESPONSIBLE  INDIVIDUAL 

Roy  R.  Wood,  Jr. 


00  FORM  1473,  83  APR  EDITION  OP  1  JAN  73  IS  OBSOLETE. 


22b.  TELEPHONE  NUMBER 
(Include  Araa  Code) 

513-255-4707 


22c.  OFFICE  SYMBOL 

AFIT/LSQ 


UNCLASSIFIED _ 

SECURITY  CLASSIFICATION  OP  THIS  PAGE 


-r, 

<v 

& 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OP  THIS  PAGE 


This  research  studied  the  development  of  source  selec¬ 
tion  evaluation  criteria  and  standards  for  reliability  and 
maintainability.  The  data  base  consisted  of  information 
obtained  during  personal  interviews  with  personnel  from  Air 
Force  Systems  Command,  Aeronautical  Systems  Division  and 
the  Air  Force  Acquisition  Logistics  Center.  Those 
interviewed  were  experienced  in  the  development  of  source 
selection  criteria  and  standards  for  reliability  and 
maintainability. 

The  research  culminated  in  the  establishment  of  a 
guide  for  the  development  of  source  selection  evaluation 
criteria  and  standards  for  reliability  and  maintainability. 
The  guide  is  not  specific  to  a  particular  type  of  system 
or  phase  in  the  acquisition  process.  The  guide  provides 
general  procedures  and  areas  of  consideration  for  develop¬ 
ment  of  criteria  and  standards  which  may  be  applied  to 
various  acquisitions. 


UNCLASSIFIED 


